{"id":128,"date":"2019-05-10T03:16:54","date_gmt":"2019-05-10T01:16:54","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/quick-and-easy-deep-learning-for-nlp-with-fast-ai\/"},"modified":"2020-06-25T17:35:44","modified_gmt":"2020-06-25T15:35:44","slug":"quick-and-easy-deep-learning-for-nlp-with-fast-ai","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/machine-learning\/quick-and-easy-deep-learning-for-nlp-with-fast-ai\/","title":{"rendered":"Quick and easy deep learning for NLP with fastai"},"content":{"rendered":"

N<\/span>owadays, Natural Language Processing<\/a> (NLP) is one of the most important fields of research<\/span>. It has seen a phenomenal rise in interest in the last few years. The basics of NLP are widely known and easy to master, but things start to get complex when the text corpus becomes huge. That\u2019s where deep learning<\/a> comes in help. In the last decade, deep learning<\/span> has proven its usefulness in computer vision tasks like image detection, classification and segmentation, but NLP applications, like text classification or machine translation, have long been considered fit for traditional machine learning<\/span> techniques only. That\u2019s not true anymore. And thanks to the amazing fastai library<\/strong>, you can quickly leverage all the most recent findings.<\/p>\n

Deep Learning frameworks and NLP<\/h2>\n

Applying deep learning to text is relatively new, with the first amazing state-of-the-art results dating back to late 2017. and some breakthrough discoveries and new techniques have been introduced in 2018, marking the ImageNet moment for NLP<\/em>, as defined by NLP researcher Sebastian Ruder<\/a>. Research<\/span> is ongoing and very fast-paced, continuously producing new state-of-the-art results almost every month. The long reign of word vectors as NLP\u2019s core representation technique has seen an exciting new line of challengers emerge: Transformer<\/a>, ELMo<\/a>, ULMFiT<\/a>, BERT<\/a> and Transformer-XL<\/a>. This is a very interesting moment in time to start exploring, studying and applying Machine Learning<\/span> to textual data<\/span>. No way to get bored.<\/p>\n

To design and train deep learning neural networks, you can choose between many frameworks, such as PyTorch<\/a> or TensorFlow<\/a>, just to name the most famous. If you don\u2019t need to deal with low level details, there are higher level libraries, built on top of them, which provide building blocks to design and train custom neural networks quickly. As Keras<\/a> is for TensorFlow, there is the fastai library<\/strong> for PyTorch.<\/p>\n

The fastai library goal is to make the training of deep neural networks as easy as possible and, at the same time, make it fast and accurate, using modern best practices<\/span>.<\/q><\/p>\n

It’s based on research<\/span> into deep learning best practices undertaken at fast.ai<\/a>, a San Francisco-based research institute founded and lead by Jeremy Howard<\/a> and Rachel Thomas<\/a>.<\/p>\n

The library includes “out of the box” support for computer vision task<\/strong>, text and natural language processing<\/span><\/strong>, tabular\/structured data classification or regression<\/strong> and collaborative filtering<\/strong> models, those at the core of modern recommendation engines.<\/p>\n

\"\"
\nWhat you can do with fastai library<\/em><\/p>\n

NLP with fastai library<\/h2>\n

The fastai library<\/strong> was the first library to make NLP Deep Learning<\/span> easy to adopt, with state-of-the-art results back in early 2018, introducing ULMFiT<\/a> (Universal Language Model Fine-tuning). It mixed various NLP deep learning techniques and machine learning<\/span> architectures derived from previous attempts and research papers, in a new, quite simple end-to-end<\/span> pipeline, with two main purposes: to deal with NLP problems where there is not so much annotated data<\/span> available and\/or big computational resources and to make NLP classification task easier.<\/p>\n

At the core, they have been able to successfully implement transfer learning and fine-tuning for textual data<\/span><\/strong>. Transfer learning is a machine learning<\/span> method where a model developed for a task is reused as the starting point for a model on a second task. Until a few years ago, it has been, and still is, a popular approach to deep learning<\/span> for computer vision tasks, where pre-trained models, such as ImageNet or Microsoft COCO, are used as the starting point, given the vast compute and time resources required to develop neural network models on these problems and from the huge jumps in skill and accuracy that they provide on related, more domain-specific problems. One common way to do this is by fine-tuning<\/em> the original model. Because the fine-tuned model doesn\u2019t have to learn from scratch, it can generally reach higher accuracy with much less data<\/span> and computation time than models that don\u2019t use transfer learning. Until then, the problem to apply this technique to textual data<\/span> was that there was a lack of knowledge<\/span> of how to train pre-trained models effectively<\/em> and this has been hindering wider adoption. This idea has been tried before, but required millions of documents<\/span> for adequate performance<\/span>. fast.ai researchers applied a variant of Stephen Merity\u2019s Language Model<\/a> (Salesforce Research), combined with other training and regularisation strategies, obtaining practical transfer learning for text applications.<\/p>\n

\"\"
\nTransfer learning<\/em><\/p>\n

In general, a language model is an NLP model which learns to predict the next word in a sentence<\/em>. It seems a trivial task, but this is very important, because for a language model to be good at guessing what someone says next, it needs a lot of world knowledge<\/em> and a deep understanding of grammar, semantics and other elements of natural language. This is exactly the kind of knowledge<\/span> that we, as humans<\/span>, leverage implicitly when we read and classify a document<\/span>. Once a language model is trained on a very big text corpus, for example Wikipedia<\/span> (as done by the fast.ai team<\/span>, making it available for everyone to use), we can then adapt it to perform other tasks, such as classifying a document<\/span>, or estimating the sentiment of a phrase.<\/p>\n

As previously stated, the fastai library<\/strong> provides some high-level APIs<\/span> and objects to quickly and easily train models on your own datasets<\/span>, quite similar for each supported data type (images, text, tabular data<\/span> or collaborative filters).<\/p>\n

Without entering in the details, for example, these few lines of code taken from the fastai library<\/strong> documentation let you train a text classifier (sentiment predictor) in few minutes, with a good level of accuracy, even if your dataset<\/span> is not so big, thanks to transfer learning<\/span> and the pre-trained English model based on Wikipedia<\/span>.<\/p>\n