Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ Neural Networksarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
Neural Networks
Article
Data sources: UnpayWall
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
arXiv.org e-Print Archive
Other literature type . Preprint . 2021
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO; Neural Networks
Other literature type . Article . 2021 . Peer-reviewed
License: Elsevier TDM
https://doi.org/10.48550/arxiv...
Article . 2021
License: arXiv Non-Exclusive Distribution
Data sources: Datacite
versions View all 9 versions
addClaim

This Research product is the result of merged Research products in OpenAIRE.

You have already added 0 works in your ORCID record related to the merged Research product.

Continual learning for recurrent neural networks: An empirical evaluation

Authors: Cossu, Andrea; Carta, Antonio; Lomonaco, Vincenzo; Bacciu, Davide;

Continual learning for recurrent neural networks: An empirical evaluation

Abstract

Learning continuously during all model lifetime is fundamental to deploy machine learning solutions robust to drifts in the data distribution. Advances in Continual Learning (CL) with recurrent neural networks could pave the way to a large number of applications where incoming data is non stationary, like natural language processing and robotics. However, the existing body of work on the topic is still fragmented, with approaches which are application-specific and whose assessment is based on heterogeneous learning protocols and datasets. In this paper, we organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks. We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications. We also provide a broad empirical evaluation of CL and Recurrent Neural Networks in class-incremental scenario, by testing their ability to mitigate forgetting with a number of different strategies which are not specific to sequential data processing. Our results highlight the key role played by the sequence length and the importance of a clear specification of the CL scenario.

Comment: Published in Neural Networks

Country
Italy
Subjects by Vocabulary

Microsoft Academic Graph classification: Computer science Continual learning Machine learning computer.software_genre Sequence Forgetting business.industry Robotics Recurrent neural network Categorization Key (cryptography) Sequential data Artificial intelligence business computer

Keywords

FOS: Computer and information sciences, Computer Science - Machine Learning, Neural Networks, Computer Science - Artificial Intelligence, Cognitive Neuroscience, Benchmarks; Continual learning; Evaluation; Recurrent neural networks; Machine Learning; Natural Language Processing; Neural Networks, Computer; Robotics, Machine Learning (cs.LG), Machine Learning, Computer, Artificial Intelligence, Evaluation, Natural Language Processing, Benchmarks, Robotics, Artificial Intelligence (cs.AI), Recurrent neural networks, continual learning; recurrent neural networks, Continual learning, Neural Networks, Computer

68 references, page 1 of 7

[7] Giuseppe Amato, Davide Bacciu, Stefano Chessa, Mauro Dragone, Claudio Gallicchio, Claudio Gennaro, Hector Lozano, Alessio Micheli, Gregory M. P. O'Hare, Arantxa Renteria, and Claudio Vairo. A benchmark dataset for human activity recognition and ambient assisted living. In Helena Lindgren, Juan F. De Paz, Paulo Novais, Antonio Fernandez-Caballero, Hyun Yoe, Andres Jimenez Ram rez, and Gabriel Villarrubia, editors, Ambient Intelligence- Software and Applications { 7th International Symposium on Ambient Intelligence (ISAmI 2016), pages 1{9. Springer International Publishing, 2016.

[8] Bernard Ans, Stephane Rousset, Robert M. French, and Serban Musca. Self-refreshing memory in arti cial neural networks: Learning temporal sequences without catastrophic forgetting. Connection Science, 16(2):71{ 99, 2004.

[9] Bernard Ans, Stephane Rousset, Robert M. French, and Serban C. Musca. Preventing Catastrophic Interference in MultipleSequence Learning Using Coupled Reverberating Elman Networks. In Proceedings of the 24th Annual Conference of the Cognitive Science Society, 2002.

[10] Nabiha Asghar, Lili Mou, Kira A Selby, Kevin D Pantasdo, Pascal Poupart, and Xin Jiang. Progressive Memory Banks for Incremental Domain Adaptation. In International Conference on Learning Representations, 2019.

[11] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015, 2014.

[12] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Je Clune, and Nick Cheney. Learning to Continually Learn. In ECAI, 2020.

[13] Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussa. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523{6541, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics.

[14] Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169{214, Copenhagen, Denmark, 2017. Association for Computational Linguistics.

[15] Johannes Buchner. Synthetic Speech Commands: A public dataset for single-word speech recognition. Kaggle Dataset, 2017.

[27] Yuwei Cui, Subutai Ahmad, and Je Hawkins. Continuous Online Sequence Learning with an Unsupervised Neural Network Model. Neural Computation, 28(11):2474{2504, 2016.

  • BIP!
    Impact byBIP!
    citations
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    49
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Top 1%
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Top 10%
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Top 1%
    OpenAIRE UsageCounts
    Usage byUsageCounts
    visibility views 50
    download downloads 67
  • citations
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    49
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Top 1%
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Top 10%
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Top 1%
    Powered byBIP!BIP!
  • 50
    views
    67
    downloads
    Powered byOpenAIRE UsageCounts
Powered by OpenAIRE graph
Found an issue? Give us feedback
visibility
download
citations
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
views
OpenAIRE UsageCountsViews provided by UsageCounts
downloads
OpenAIRE UsageCountsDownloads provided by UsageCounts
49
Top 1%
Top 10%
Top 1%
50
67
Green
bronze
Funded by