publication . Preprint . 2019

Exchangeable Generative Models with Flow Scans

Bender, Christopher; O'Connor, Kevin; Li, Yang; Garcia, Juan Jose; Zaheer, Manzil; Oliva, Junier;
Open Access English
  • Published: 05 Feb 2019
Abstract
In this work, we develop a new approach to generative density estimation for exchangeable, non-i.i.d. data. The proposed framework, FlowScan, combines invertible flow transformations with a sorted scan to flexibly model the data while preserving exchangeability. Unlike most existing methods, FlowScan exploits the intradependencies within sets to learn both global and local structure. FlowScan represents the first approach that is able to apply sequential methods to exchangeable density estimation without resorting to averaging over all possible permutations. We achieve new state-of-the-art performance on point cloud and image set modeling.
Subjects
free text keywords: Computer Science - Machine Learning, Statistics - Machine Learning
Funded by
NIH| Big Data to Knowledge Training Program
Project
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5T32LM012420-04
  • Funding stream: NATIONAL LIBRARY OF MEDICINE
Download from
21 references, page 1 of 2

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pp. 265-283, 2016.

Bernardo, J. M. and Smith, A. F. Bayesian theory, volume 405. John Wiley & Sons, 2009.

Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation. CoRR, abs/1410.8516, 2014.

Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. CoRR, abs/1605.08803, 2016. [OpenAIRE]

Edwards, H. and Storkey, A. Towards a neural statistician. In 5th International Conference on Learning Representations (ICLR 2017), 2 2017.

Germain, M., Gregor, K., Murray, I., and Larochelle, H. MADE: Masked Autoencoder for Distribution Estimation. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 881-889, Lille, France, 07-09 Jul 2015. PMLR.

Ghahramani, Z. and Heller, K. A. Bayesian sets. In Weiss, Y., Scho¨lkopf, B., and Platt, J. C. (eds.), Advances in Neural Information Processing Systems 18, pp. 435-442. MIT Press, 2006.

Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep autoregressive networks. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1242-1250, Bejing, China, 22-24 Jun 2014. PMLR.

Heller, K. A. and Ghahramani, Z. A simple bayesian framework for content-based image retrieval. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 2, pp. 2110-2117. IEEE, 2006.

Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10236-10245, 2018.

Korshunova, I., Degrave, J., Huszar, F., Gal, Y., Gretton, A., and Dambre, J. Bruno: A deep recurrent model for exchangeable data. In Advances in Neural Information Processing Systems, pp. 7190-7198, 2018.

Larochelle, H. and Murray, I. The neural autoregressive distribution estimator. In Gordon, G., Dunson, D., and Dudk, M. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 29-37, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. [OpenAIRE]

Oliva, J. B., Dubey, A., Po´czos, B., Schneider, J., and Xing, E. P. Transformation autoregressive networks. arXiv preprint arXiv:1801.09819, 2018.

Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017.

Rezatofighi, S. H., Milan, A., Abbasnejad, E., Dick, A., Reid, I., et al. Deepsetnet: Predicting sets with deep neural networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 5257-5266. IEEE, 2017.

21 references, page 1 of 2
Related research
Abstract
In this work, we develop a new approach to generative density estimation for exchangeable, non-i.i.d. data. The proposed framework, FlowScan, combines invertible flow transformations with a sorted scan to flexibly model the data while preserving exchangeability. Unlike most existing methods, FlowScan exploits the intradependencies within sets to learn both global and local structure. FlowScan represents the first approach that is able to apply sequential methods to exchangeable density estimation without resorting to averaging over all possible permutations. We achieve new state-of-the-art performance on point cloud and image set modeling.
Subjects
free text keywords: Computer Science - Machine Learning, Statistics - Machine Learning
Funded by
NIH| Big Data to Knowledge Training Program
Project
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5T32LM012420-04
  • Funding stream: NATIONAL LIBRARY OF MEDICINE
Download from
21 references, page 1 of 2

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pp. 265-283, 2016.

Bernardo, J. M. and Smith, A. F. Bayesian theory, volume 405. John Wiley & Sons, 2009.

Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation. CoRR, abs/1410.8516, 2014.

Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. CoRR, abs/1605.08803, 2016. [OpenAIRE]

Edwards, H. and Storkey, A. Towards a neural statistician. In 5th International Conference on Learning Representations (ICLR 2017), 2 2017.

Germain, M., Gregor, K., Murray, I., and Larochelle, H. MADE: Masked Autoencoder for Distribution Estimation. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 881-889, Lille, France, 07-09 Jul 2015. PMLR.

Ghahramani, Z. and Heller, K. A. Bayesian sets. In Weiss, Y., Scho¨lkopf, B., and Platt, J. C. (eds.), Advances in Neural Information Processing Systems 18, pp. 435-442. MIT Press, 2006.

Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep autoregressive networks. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1242-1250, Bejing, China, 22-24 Jun 2014. PMLR.

Heller, K. A. and Ghahramani, Z. A simple bayesian framework for content-based image retrieval. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 2, pp. 2110-2117. IEEE, 2006.

Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10236-10245, 2018.

Korshunova, I., Degrave, J., Huszar, F., Gal, Y., Gretton, A., and Dambre, J. Bruno: A deep recurrent model for exchangeable data. In Advances in Neural Information Processing Systems, pp. 7190-7198, 2018.

Larochelle, H. and Murray, I. The neural autoregressive distribution estimator. In Gordon, G., Dunson, D., and Dudk, M. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 29-37, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. [OpenAIRE]

Oliva, J. B., Dubey, A., Po´czos, B., Schneider, J., and Xing, E. P. Transformation autoregressive networks. arXiv preprint arXiv:1801.09819, 2018.

Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017.

Rezatofighi, S. H., Milan, A., Abbasnejad, E., Dick, A., Reid, I., et al. Deepsetnet: Predicting sets with deep neural networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 5257-5266. IEEE, 2017.

21 references, page 1 of 2
Related research
Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue