{"id":8576,"date":"2020-09-07T11:34:33","date_gmt":"2020-09-07T09:34:33","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=8576"},"modified":"2022-01-05T20:05:53","modified_gmt":"2022-01-05T19:05:53","slug":"machine-learning-edge-example","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/machine-learning\/machine-learning-edge-example\/","title":{"rendered":"ML at the Edge: a Practical Example"},"content":{"rendered":"\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tTable Of Contents\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. Gesture recognition<\/a>
  2. Techniques for gesture recognition<\/a>
  3. Machine learning for gesture recognition<\/a>
  4. A practical example of edge ML<\/a>
  5. Running the model<\/a>
  6. Conclusions and what next<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n

    Machine learning<\/span> is the primary methodology<\/span> for delivering AI applications<\/span>. In previous articles<\/span>, I discussed the main reasons behind moving machine learning to the network edge<\/a>. These include the need for real-time performance<\/span>, security<\/span> considerations, and a lack of connectivity. However, ML models typically require powerful hardware<\/span> to run efficiently. Fortunately, in recent years, MCUs have become ever more powerful, alongside the release of new frameworks enabling edge ML. TensorFlow Lite, in particular, has gained significant traction.<\/p>\n\n\n\n

    The last article<\/a> discussed choosing a suitable platform for your edge ML project<\/span>. It also explored some of the limitations imposed by TensorFlow Lite<\/strong>. In this article<\/span>, we look at a practical example of running a TensorFlow Lite model<\/span> on an NXP i.MX RT1050 EVK<\/a>. Specifically, I\u2019ll show how you can perform gesture<\/span> recognition using the built-in accelerometers and a simple neural network.<\/p>\n\n\n\n

    Gesture recognition<\/h2>\n\n\n\n

    Gesture recognition<\/span> is a growing field in human-computer interaction<\/span> using a combination of sensors<\/span> and algorithms to identify specific human gestures. It enables many HCI<\/span> applications<\/span> ranging from controlling your car stereo to interactive<\/span> gaming<\/span>. It can even identify individuals<\/a> based on the different ways they hold and use a device<\/span>. <\/p>\n\n\n\n

    Applications of gesture recognition<\/h3>\n\n\n\n

    One of the earliest well-known applications<\/span> of gesture<\/span> recognition was the Nintendo<\/span> Wii<\/span>. The Wii<\/span> used hand-held<\/span> accelerometers enabling players to interact with various virtual sports events. It was only able to detect large gestures, but that was ideal for playing virtual sports like tennis or golf.<\/p>\n\n\n\n

    \"\"
    The Nintendo Wii offered simple controllers with accelerometers to enable gesture recognition<\/figcaption><\/figure><\/div>\n\n\n\n

    More recently, we have seen a rush to install gesture recognition as a safety feature in cars. Many motoring accidents are caused by the driver being distracted, one of the most frequent distractions being the audio system<\/span>. Gesture recognition<\/span> allows the driver to control the system<\/span> without taking their eyes off the road: waving in a specific pattern<\/span> to skip a track, increase volume, or change channel. <\/p>\n\n\n\n

    Techniques for gesture recognition<\/h2>\n\n\n\n

    There are three broad forms of gesture<\/span> recognition. The first relies on detecting movement in a sensor<\/span>\u2019s field of view<\/span>. The second uses accelerometers<\/span> within a device<\/span> being held or worn by the user<\/span>. The third uses computer vision to process the input from a camera.<\/p>\n\n\n\n

    IR sensors<\/h3>\n\n\n\n

    IR<\/span> movement detection has existed for decades and is still widely used in burglar alarm systems, for example. However, this sort of movement detection is binary. That is to say, either it detects movement or it doesn\u2019t, which is not sufficient to enable gesture<\/span> recognition. Instead, a time of flight sensor<\/a> is required. These send out pulses of IR<\/span> energy and measure the time for the reflected pulse to return, thus identifying simple gestures.<\/p>\n\n\n\n

    Accelerometers<\/h3>\n\n\n\n

    Our world is full of accelerometers<\/span>: in mobile<\/span> phones, in video game<\/span> controllers<\/span>, even in TV remote<\/span> controls. Modern accelerometers<\/span> are typically 6 axis, and often incorporate other sensors<\/span>, providing data<\/span> on movement in the XYZ directions, along with pitch, yaw, and roll. All of which can be used to determine gestures<\/span>.<\/p>\n\n\n\n

    \"\"<\/figure><\/div>\n\n\n\n

    Computer vision<\/h3>\n\n\n\n

    The most powerful approach for gesture<\/span> recognition involves the application<\/span> of computer vision. Here, you train a set of ML models to identify a hand in a video, track its movement, and recognise gestures<\/span>. This approach has two key advantages over the other techniques: it doesn\u2019t require the user<\/span> to be holding a physical object (as is the case with accelerometers<\/span>), and it can work at much longer ranges than systems<\/span> based on time of flight sensors<\/span>. <\/p>\n\n\n\n

    Time of flight cameras<\/h3>\n\n\n\n

    One of the newest approaches for gesture<\/span> recognition takes advantage of time-of-flight (ToF) cameras.<\/a> These devices<\/span> combine a camera with the ability to measure the distance to an object. They provide the ability to create 3D models<\/span> of the environment without the need for expensive LIDAR scanning technology. As a result, they are perfect for recognising complex gestures<\/span> at a distance.<\/p>\n\n\n\n

    Machine learning for gesture recognition<\/h2>\n\n\n\n

    Machine learning<\/span> is one of the most flexible techniques for enabling gesture<\/span> recognition. It allows the identification of complex gestures<\/span> yet is agnostic to the person making the gesture<\/span>. ML models can take inputs from multiple different sensors<\/span> and can cope with noisy data, outperforming the purely algorithmic approach taken by systems<\/span> like Nintendo<\/span> Wii<\/span> or in-car audio. So, how do you create a machine learning<\/span> model<\/span> for gesture<\/span> recognition and port it to your edge device?<\/p>\n\n\n\n

    A practical example of edge ML<\/h2>\n\n\n\n

    The previous article<\/a> explained the steps needed to create a TensorFlow Lite model<\/span> and port it to a device<\/span>. Now let\u2019s see what this looks like in practice using a simple gesture<\/span> recognition model<\/span>. <\/p>\n\n\n\n

    This example uses the  available from  and one of the most powerful MCU<\/span> boards in its class<\/span>. It is based on an ARM<\/span> Cortex 7 processor running at up to 600MHz. The board specification<\/span> includes:<\/p>\n\n\n\n