publication . Preprint . 2017

TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow

Hafner, Danijar; Davidson, James; Vanhoucke, Vincent;
Open Access English
  • Published: 08 Sep 2017
Abstract
We introduce TensorFlow Agents, an efficient infrastructure paradigm for building parallel reinforcement learning algorithms in TensorFlow. We simulate multiple environments in parallel, and group them to perform the neural network computation on a batch rather than individual observations. This allows the TensorFlow execution engine to parallelize computation, without the need for manual synchronization. Environments are stepped in separate Python processes to progress them in parallel without interference of the global interpreter lock. As part of this project, we introduce BatchPPO, an efficient implementation of the proximal policy optimization algorithm. By...
Subjects
free text keywords: Computer Science - Machine Learning, Computer Science - Artificial Intelligence
Download from

[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensor ow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.

[2] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.

[3] L. Dalcin. Mpi for python: Python bindings for mpi. https://github.com/ mpi4py/mpi4py, 2017.

[4] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 1329{1338, 2016.

[5] R. H. Hahnloser, R. Sarpeshkar, M. A. Mahowald, R. J. Douglas, and H. S. Seung. Digital selection and analogue ampli cation coexist in a cortex-inspired silicon circuit. Nature, 405(6789):947, 2000. [OpenAIRE]

[6] N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, A. Eslami, M. Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017. [OpenAIRE]

[7] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[8] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928{1937, 2016.

[9] J. Schulman. Modular rl: Implementation of trpo and related algorithms. https://github.com/joshu/modular_rl, 2017.

[10] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[11] S. Sidor, J. Schulman, M. Plappert, and contributors. Openai baselines: High-quality implementations of reinforcement learning algorithms. https: //github.com/openai/baselines, 2017.

[12] Y. Tian, Q. Gong, W. Shang, Y. Wu, and L. Zitnick. Elf: An extensive, lightweight and exible research platform for real-time strategy games. arXiv preprint arXiv:1707.01067, 2017.

[13] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for modelbased control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026{5033. IEEE, 2012.

[14] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229{256, 1992.

Abstract
We introduce TensorFlow Agents, an efficient infrastructure paradigm for building parallel reinforcement learning algorithms in TensorFlow. We simulate multiple environments in parallel, and group them to perform the neural network computation on a batch rather than individual observations. This allows the TensorFlow execution engine to parallelize computation, without the need for manual synchronization. Environments are stepped in separate Python processes to progress them in parallel without interference of the global interpreter lock. As part of this project, we introduce BatchPPO, an efficient implementation of the proximal policy optimization algorithm. By...
Subjects
free text keywords: Computer Science - Machine Learning, Computer Science - Artificial Intelligence
Download from

[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensor ow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.

[2] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.

[3] L. Dalcin. Mpi for python: Python bindings for mpi. https://github.com/ mpi4py/mpi4py, 2017.

[4] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 1329{1338, 2016.

[5] R. H. Hahnloser, R. Sarpeshkar, M. A. Mahowald, R. J. Douglas, and H. S. Seung. Digital selection and analogue ampli cation coexist in a cortex-inspired silicon circuit. Nature, 405(6789):947, 2000. [OpenAIRE]

[6] N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, A. Eslami, M. Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017. [OpenAIRE]

[7] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[8] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928{1937, 2016.

[9] J. Schulman. Modular rl: Implementation of trpo and related algorithms. https://github.com/joshu/modular_rl, 2017.

[10] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[11] S. Sidor, J. Schulman, M. Plappert, and contributors. Openai baselines: High-quality implementations of reinforcement learning algorithms. https: //github.com/openai/baselines, 2017.

[12] Y. Tian, Q. Gong, W. Shang, Y. Wu, and L. Zitnick. Elf: An extensive, lightweight and exible research platform for real-time strategy games. arXiv preprint arXiv:1707.01067, 2017.

[13] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for modelbased control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026{5033. IEEE, 2012.

[14] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229{256, 1992.

Powered by OpenAIRE Research Graph
Any information missing or wrong?Report an Issue