New model architectures: ALBERT, CamemBERT, GPT2-XL, DistilRoberta Four new models have been added in v2.2.0 ALBERT (Pytorch & TF) (from Google Research and the Toyota Technological Institute at Chicago) released with the paper ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut. CamemBERT (Pytorch) (from Facebook AI Research, INRIA, and La Sorbonne Université), as the first large-scale Transformer language model. Released alongside the paper CamemBERT: a Tasty French Language Model by Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suarez, Yoann Dupont, Laurent Romary, Eric Villemonte de la Clergerie, Djame Seddah, and Benoît Sagot. It was added by @louismartin with the help of @julien-c. DistilRoberta (Pytorch & TF) from @VictorSanh as the third distilled model after DistilBERT and DistilGPT-2. GPT-2 XL (Pytorch & TF) as the last GPT-2 checkpoint released by OpenAI Encoder-Decoder architectures We welcome the possibility to create fully seq2seq models by incorporating Encoder-Decoder architectures using a PreTrainedEncoderDecoder class that can be initialized from pre-trained models. The base BERT class has be modified so that it may behave as a decoder. Furthermore, a Model2Model class that simplifies the definition of an encoder-decoder when both encoder and decoder are based on the same model has been added. @rlouf Benchmarks and performance improvements Works by @tlkh and @LysandreJik aiming to benchmark the library models with different technologies: with TensorFlow and Pytorch, with mixed precision (AMP and FP-16) and with model tracing (Torchscript and XLA). A new section was created in the documentation: benchmarks pointing to Google sheets with the results. Breaking changes Tokenizers now add special tokens by default. @LysandreJik New model templates Model templates to ease the addition of new models to the library have been added. @thomwolf Inputs Embeddings A new input has been added to all models' forward (for Pytorch) and call (for TensorFlow) methods. These inputs_embeds are a direct embedded representation. This is useful as it gives more control over how to convert input_ids indices into associated vectors than the model's internal embedding lookup matrix. @julien-c Getters and setters for input and output embeddings A new API for the input and output embeddings are available. These methods are model-independent and allow easy acquisition/modification of the models' embeddings. @thomwolf Additional architectures New model architectures are available, namely: DistilBertForTokenClassification, CamembertForTokenClassification @stefan-it Community additions/bug-fixes/improvements The Fairseq RoBERTa model conversion script has been patched. @louismartin einsum now runs in FP-16 in the library's examples @slayton58 In-depth work on the squad script for XLNet to reproduce the original paper's results @hlums Additional improvements on the run_squad script by @WilliamTambellini, @orena1 The run_generation script has seen several improvements by @leo-du The RoBERTaTensorFlow model has been patched for several use-cases: TPU and keras.fit @LysandreJik The documentation is now versioned, links are available on the github readme @LysandreJik The run_ner script has seen several improvements @mmaybeno, @oneraghavan, @manansanghi The run_tf_glue script now works for all GLUE tasks @LysandreJik The run_lm_finetuning script now correctly evaluates perplexity on MLM tasks @altsoph An issue related to the XLM TensorFlow implementation's training has been fixed @tlkh run_bertology has been updated to be closer to the run_glue example @adrianbg Fixed added special tokens in decoded sequences @LysandreJik Several performance improvements have been done to the tokenizers @iedmrc A memory leak has been identified and patched in the library's schedulers @rlouf Correct warning when encoding a sequence too long while specifying a maximum length @LysandreJik Resizing the token embeddings now works as expected in the run_lm_finetuning script @iedmrc The difference in versions between Pypi/source in order to run the examples has been clarified @rlouf