On Variational Bounds of Mutual Information

Preprint English OPEN
Poole, Ben; Ozair, Sherjil; Oord, Aaron van den; Alemi, Alexander A.; Tucker, George;
  • Subject: Statistics - Machine Learning | Computer Science - Machine Learning

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural n... View more
  • References (51)
    51 references, page 1 of 6

    Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.

    Alemi, A. A., Poole, B., Fischer, I., Dillon, J. V., Saurous, R. A., and Murphy, K. Fixing a broken elbo, 2017.

    Barber, D. and Agakov, F. The im algorithm: A variational approach to information maximization. In NIPS, pp. 201- 208. MIT Press, 2003.

    Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Hjelm, D., and Courville, A. Mutual information neural estimation. In International Conference on Machine Learning, pp. 530-539, 2018.

    Bell, A. J. and Sejnowski, T. J. An informationmaximization approach to blind separation and blind deconvolution. Neural computation, 7(6):1129-1159, 1995.

    Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-877, 2017.

    Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding disentangling in -vae. arXiv preprint arXiv:1804.03599, 2018.

    Chen, T. Q., Li, X., Grosse, R., and Duvenaud, D. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04942, 2018.

    Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.

    Donsker, M. D. and Varadhan, S. S. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36 (2):183-212, 1983.

  • Related Research Results (1)
  • Metrics
Share - Bookmark