{"id":6507,"date":"2020-07-17T11:41:32","date_gmt":"2020-07-17T09:41:32","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=6507"},"modified":"2022-01-05T20:05:30","modified_gmt":"2022-01-05T19:05:30","slug":"the-cutting-edge-of-real-time-ai","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/the-cutting-edge-of-real-time-ai\/","title":{"rendered":"The cutting edge of real-time AI"},"content":{"rendered":"\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tTable Of Contents\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. Introduction to Machine Learning\u00a0<\/a>
  2. The drawbacks of centralised AI<\/a>
  3. The impact of moving AI to the edge<\/a>
  4. What is needed to run AI and ML models at the edge?<\/a>
  5. The trade-off\u00a0<\/a>
  6. Conclusions<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n

    Machine learning<\/a>\u00a0is traditionally processor-intensive; ML algorithms require large numbers of parallel operations. As a result, the models usually run in data centres at the core of the network. However, this has a direct impact in terms of latency, security and cost. Over the next six months, this article series will explore\u00a0why AI is moving to the network edge and the technology that\u2019s making it possible. We\u2019ll also provide practical examples of implementation on a variety of popular embedded platforms.<\/p>\n\n\n\n

    RELATED CONTENT<\/h4><\/div>

    Getting started with edge machine learning<\/a><\/p><\/div><\/div>

    <\/div><\/div><\/div><\/div>\n\n\n\n

    Introduction to Machine Learning <\/h2>\n\n\n\n

    Machine learning<\/span> lies at the heart of most AI applications<\/span>, and involves teaching a computer<\/span> to identify patterns<\/span> in data<\/span>. More specifically, the goal is to create a trained model. This can be done with supervised learning<\/span>, where the computer<\/span> is shown examples to learn from. Alternatively, the process can be unsupervised – the computer<\/span> simply looks for interesting patterns<\/span> in the data<\/span>. Techniques involving continuous or ongoing learning<\/span>, where the computer<\/span> learns from its mistakes, also exist but are outside the scope of this article.<\/p>\n\n\n\n

    Running your ML model.<\/h3>\n\n\n\n

    Once the ML model has been created, it can be used to deliver your project<\/span>. Models can be used for forecasting future events, identifying anomalies, and image or speech recognition<\/span>. In nearly all cases, models rely on large deep tree structures and need significant computing power<\/span> to run. This is especially true for models engaged in image and voice recognition<\/span>, which are usually based on artificial neural networks<\/span>. Neural networks<\/span> create dense meshes and hence need to be run on highly parallelised hardware, often based on GPUs<\/span>. Until recently, such power has only been available in the cloud<\/span> on AWS<\/span><\/a>, Google Cloud<\/span>, or Azure<\/span><\/a>.<\/p>\n\n\n\n

    To get some idea of the power required, look at the specifications of AWS P3 instances, optimised for ML.<\/p>\n\n\n\n

    Type<\/strong><\/td>GPUs<\/strong><\/td>vCPUs<\/strong><\/td>RAM<\/strong><\/td>GPU RAM<\/strong><\/td>Storage BW<\/strong><\/td>Network BW<\/strong><\/td><\/tr>
    P3.2XL<\/td>1<\/td>8<\/td>61GB<\/td>16GB<\/td>1.5Gbps<\/td>~10Gbps<\/td><\/tr>
    P3.8XL<\/td>4<\/td>32<\/td>244GB<\/td>64GB<\/td>7Gbps<\/td>10Gbps<\/td><\/tr>
    P3.16XL<\/td>8<\/td>64<\/td>488GB<\/td>128GB<\/td>14Gbps<\/td>25Gbps<\/td><\/tr>
    P3.24XL<\/td>8<\/td>96<\/td>768GB<\/td>256GB<\/td>19Gbps<\/td>100Gbps<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    As you can see, these machines are seriously powerful; they have huge amounts of RAM<\/span>, with extremely fast network and storage access. Above all, they have significant CPU<\/span> and GPU<\/span> processing power<\/span>, which makes running ML models at the network edge a real challenge.<\/p>\n\n\n\n

    The drawbacks of centralised AI<\/h2>\n\n\n\n

    To date, most well-known applications<\/span> of AI<\/span> have relied on the cloud<\/span> because it is so hard to run ML models<\/span> at the edge. However, this dependence on cloud computing<\/span> imposes some limitations on using AI<\/span>. Here is a list of some of the operational drawbacks to centralised AI<\/span>. <\/p>\n\n\n\n

    Some applications can\u2019t run in the cloud<\/h3>\n\n\n\n

    To operate AI<\/span> in the cloud<\/span>, you need to have a decent and stable network connection. As a result, there are some AI<\/span> applications<\/span> that have to run locally due to a lack of network connectivity. In other words, these applications<\/span> only work if you are able to run your ML models<\/span> at the edge. <\/p>\n\n\n\n

    An obvious example here is self-driving vehicles; these need to do a number of tasks that rely on machine learning<\/span>. The most important of these tasks is object detection<\/span> and avoidance. This requires quite demanding ML models<\/span> that need a reasonable degree of computing<\/span> power. However, even networked cars have only low bandwidth<\/span> connections, and these connections are inconsistent. <\/p>\n\n\n\n

    This limitation also applies when creating smart IoT<\/span> monitoring systems for mining and other heavy industries. There is often a fast network locally, but Internet connectivity<\/span> may be reliant on a satellite uplink.<\/p>\n\n\n\n

    Latency matters<\/h3>\n\n\n\n

    Many ML applications<\/span> need to work in real-time. Self-driving cars<\/span>, as mentioned above, are such an application<\/span>, but there are also applications<\/span> such as real-time facial recognition. This can be used for door entry systems or for security<\/span> purposes; for example, police forces often use this technology<\/span> to monitor football crowds in an attempt to identify known trouble-makers. <\/p>\n\n\n\n

    AI is also increasingly being used to create smart medical devices. Some of these need to work in real-time to deliver real benefits, but the average round trip time to connect to a data<\/span> centre is in the order of 10-100ms. Real-time<\/span> applications<\/span> are therefore hard to achieve without moving your ML models nearer to the network edge.<\/p>\n\n\n\n

    Security may be an issue<\/h3>\n\n\n\n

    A lot of ML applications<\/span> deal with secure or sensitive data<\/span>. It is clearly possible to send this data<\/span> across the network, and store it in the cloud<\/span>, securely. However, local policies often forbid that. The obvious example here is medical AI applications<\/span>. <\/p>\n\n\n\n

    Health data<\/span> is especially sensitive, and there are strict rules in many countries about sending it to a cloud server. Overall, it is always easier to ensure the security<\/span> of a device that is only connected to a local network. <\/p>\n\n\n\n

    Costs<\/h3>\n\n\n\n

    Running ML in the cloud<\/span> does not come cheap. For a start, ML-optimised cloud instances are pretty expensive – the lowest spec instance<\/span> in the table given above costs<\/span> about US$3 an hour. Then all the extra costs<\/span> cloud<\/span> providers charge need to be considered, such as fees for storage can and network access. Realistically, running an AI application<\/span> could easily end up costing US$3,000 a month.<\/p>\n\n\n\n

    The impact of moving AI to the edge<\/h2>\n\n\n\n

    As we have seen, there are some strong arguments for moving ML models to the edge of the network, but which AI applications<\/span> really benefit from doing so? The list of drawbacks above gives some strong hints. Check whether any of the following apply to the project<\/span>:<\/p>\n\n\n\n