Open-ended systems and unknown dynamical environments present challenges to the traditional machine learning systems, and in many cases traditional methods are not applicable. Lifelong reinforcement learning is a special case of dynamic (process-oriented) reinforcement learning. Multi-task learning is a methodology that exploits similarities and patterns across multiple tasks. Both can be successfully used for open-ended systems and automated learning in unknown environments. Due to its unique characteristics, lifelong reinforcement presents both challenges and potential capabilities that go beyond traditional reinforcement learning methods. In this article, we present the basic notions of lifelong reinforcement learning, introduce the main methodologies, applications and challenges. We also introduce a new model of lifelong reinforcement based on the Evolvable Virtual Machine architecture (EVM).