{"id":11239,"date":"2020-10-12T09:52:24","date_gmt":"2020-10-12T07:52:24","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=11239"},"modified":"2022-01-05T20:06:17","modified_gmt":"2022-01-05T19:06:17","slug":"image-recognition-on-mcu","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/image-recognition-on-mcu\/","title":{"rendered":"Seeing Is Believing: Image Recognition on a \u20ac10 MCU"},"content":{"rendered":"\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tTable Of Contents\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. Why we need image recognition at the edge<\/a>
  2. A brief history of image recognition<\/a>
  3. Using CNNs for image recognition<\/a>
  4. Steps for image recognition<\/a>
  5. A practical implementation<\/a>
  6. Conclusions<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n

    This series is exploring the rationale for moving machine learning<\/span> to the network edge. This article<\/span> looks in more detail at image<\/span> recognition, one of the prime use cases<\/span> for ML<\/span> at the edge.<\/p>\n\n\n\n

    As explained in previous articles<\/span>, there are many use cases<\/span> for machine learning at the network edge<\/a>. So far, I explained the rationale for running ML models at the edge. introduced some of the hardware and tools<\/a>, and gave a practical example of implementing gesture recognition<\/a>. Here, I explore image<\/span> recognition in detail. You will learn what image<\/span> recognition is and how convolutional neural networks<\/span> help implement it. At the end, there is a practical example of implementing image<\/span> recognition on a Microchip<\/span> SAM E54 MCU<\/span>.<\/p>\n\n\n\n

    Why we need image recognition at the edge<\/h2>\n\n\n\n

    Nowadays, everyone is familiar with image<\/span> recognition. It enables autonomous driving. It powers facial recognition systems<\/span>. Medics can even use it to diagnose breast cancer<\/a> from mammograms. Many of these applications<\/span> need to run in real-time<\/span>. They cannot rely on network connectivity and often they need to run in lightweight hardware<\/span> with low power draw. As a result, they are a perfect use case<\/span> for ML<\/span> at the network edge.<\/p>\n\n\n\n

    All these applications<\/span> rely on ML models, such as convolutonal neural networks<\/a> (CNNs). As explained below, these algorithms allow a computer to pick out and identify features<\/span> within an image<\/span>. However, CNNs are complex, requiring lots of parallel operations<\/span> to run efficiently. Fortunately, modern embedded MCUs are able to run quite large neural networks<\/span>. But let\u2019s start by looking at the history of CNNs and image<\/span> recognition.<\/p>\n\n\n\n

    A brief history of image recognition<\/h2>\n\n\n\n

    Image recognition<\/span> may seem quite new. It\u2019s only a few years since the first reports about computers<\/span> learning<\/span> to recognise cats. But in fact, image<\/span> recognition can be traced back many decades. Of course, the original image<\/span> recognition systems were nothing like as powerful as today. But they were still able to perform some useful tasks.<\/p>\n\n\n\n

    The artificial neuron<\/h3>\n\n\n\n

    The underlying element in any neural network<\/span> is called an artificial neuron. These data structures are loosely based on human neurons. Each neuron takes a number of weighted<\/span> inputs and combines them using a transfer function<\/span>. It then uses an activation function<\/span> to determine whether to fire or not. This is shown in the following diagram.<\/p>\n\n\n\n

    \"Artificial
    Structure of an artificial neural network<\/figcaption><\/figure><\/div>\n\n\n\n

    The transfer function<\/span> is typically a simple sum or product. The activation function<\/span> can be one of several operators including a step, identity function<\/span>, or logistic function<\/span>.<\/p>\n\n\n\n

    \"Different
    Different functions give different outputs from the neuron<\/figcaption><\/figure><\/div>\n\n\n\n

    If the output of the activation function<\/span> exceeds the threshold, the neuron fires, otherwise it remains dormant.<\/p>\n\n\n\n

    From neurons to neural networks<\/h3>\n\n\n\n

    By itself, an artificial neuron is not much use. But they are very powerful when you combine them into neural networks<\/span>. All artificial neural networks (ANNs) consist of at least 3 layers: an input layer<\/span>, 1 or more hidden layers, and an output layer<\/span>. The number of inputs depends on your data<\/span>. The ANN classifies<\/span> the input data<\/span> into 2 or more outputs. When you feed in your data<\/span>, only one output neuron should activate. Which neuron fires depends on the weights<\/span> you set in the network. The process<\/span> of setting these weights<\/span> so that the correct output fires is called training<\/span>. <\/p>\n\n\n\n

    Using CNNs for image recognition<\/h2>\n\n\n\n

    Convolutional neural networks<\/span> are a form of deep neural network<\/span> widely used for image<\/span> recognition. They are deep because they utilise multiple hidden layers<\/span>. They are convolutional because many of their hidden layers<\/span> convolve<\/span> or simplify the input image<\/span> into a feature map. For instance<\/span>, you might take a 64×64 pixel<\/span> bitmap<\/span> and reduce it to a 4×4 matrix showing which areas of the image<\/span> were more dark and which more light.<\/p>\n\n\n\n

    \"Convolution
    Convolution can simplify hand-written numerals<\/figcaption><\/figure><\/div>\n\n\n\n

    One of the fathers of image<\/span> recognition is the French computer scientist<\/span>, Yann LeCun<\/a>. In the late 1980s, he worked at AT&T Bell Laboratories in New Jersey. A key project at the lab looked to create a system<\/span> to recognise hand-written zip codes on envelopes. The aim was to automate the sorting of mail. In 1989, LeCun\u2019s colleagues published a paper showing how to use a neural network<\/span> to perform this task. However, this network had to be laboriously tuned by hand. LeCun made a huge breakthrough by applying a process<\/span> called backpropagation<\/a> to the problem. <\/p>\n\n\n\n

    Backpropagation<\/span> involves taking the outputs and working backwards through the CNN changing the weights<\/span> of each neuron, the aim being to reduce the mean error each time. There are numerous approaches for doing this, such as gradient descent. You can read more about it in this tutorial<\/a> on Towards Data Science.<\/p>\n\n\n\n

    Steps for image recognition<\/h2>\n\n\n\n

    Modern image<\/span> recognition divides the problem into three steps. Detection (or localisation), classification<\/span>, and segmentation<\/span>. <\/p>\n\n\n\n

    Detection\/localisation<\/h3>\n\n\n\n

    This involves identifying different features<\/span> within the image<\/span>. In the figure below (from a Facebook Engineering blogpost<\/a>) the system<\/span> has identified a number of different elements.<\/p>\n\n\n\n

    \"Facebook
    Facebook engineers demonstrate how object detection works<\/figcaption><\/figure><\/div>\n\n\n\n

    There are numerous approaches to detection and localisation. The aim is to identify whether adjacent pixels<\/span> are related to each other. In the handwritten numeral 3 example above, this is relatively easy. You just identify which areas are black and which are white. But in a detailed colour image<\/span>, this gets more tricky.<\/p>\n\n\n\n

    Classification<\/h3>\n\n\n\n

    Next, you try to classify each region you have identified. This is where machine learning<\/span> really comes to the fore. It allows you to train your classifier<\/span> against a large, labelled dataset<\/span>. In the image<\/span> above, the classifier<\/span> is able to tell that there are 3 classes of object<\/span>. It is thus able to determine that the image<\/span> contains a man, a dog, and 5 sheep. <\/p>\n\n\n\n

    Segmentation<\/h4>\n\n\n\n

    The final stage is to understand how the items relate to each other. For instance<\/span>, is the man in front of or behind the sheep? This process<\/span> is known as segmentation<\/span> or semantic image<\/span> segmentation<\/span>. The image<\/span> below shows the result for the picture<\/span> of the shepherd.<\/p>\n\n\n\n

    \"The
    The system is able to segment the image and work out the relationship between elements<\/figcaption><\/figure><\/div>\n\n\n\n

    This gets much harder when you have complex images<\/span> like a typical street scene. Self-driving vehicles need to do real-time segmentation<\/span> as shown in the following frame taken from Hengshuang Zhao\u2019s YouTube video \u2018ICNet for Real-Time Semantic Segmentation<\/span> on High-Resolution Images\u2019.<\/p>\n\n\n\n

    \"There
    There are many different objects to interpret in this street scene<\/figcaption><\/figure><\/div>\n\n\n\n

    Once a neural network<\/span> has been trained<\/span> to perform image<\/span> recognition, you can deploy it on any suitable hardware<\/span>. Here, we are interested in deploying image<\/span> recognition at the network edge. The rest<\/span> of this article<\/span> explores how to do this in practice.<\/p>\n\n\n\n

    A practical implementation<\/h2>\n\n\n\n

    As we already saw, image<\/span> recognition requires a trained<\/span> neural network or other machine learning model. In this example, we will create a simple person-detection model. For this, you need the model, a platform to run the model on, and a camera or other image source. For this example, I have selected the Microchip SAM E54. Or, more precisely, the SAM E54 Xplained Pro Evaluation Kit.<\/a> This MCU evaluation board is ideal for developing ML models:<\/p>\n\n\n\n