{"id":12411,"date":"2021-01-13T09:00:00","date_gmt":"2021-01-13T08:00:00","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=12411"},"modified":"2022-01-05T20:03:23","modified_gmt":"2022-01-05T19:03:23","slug":"state-of-ai-2021","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/state-of-ai-2021\/","title":{"rendered":"The State of AI in 2021"},"content":{"rendered":"\r\n

We know that technology<\/span> tends to move in hype cycles, where certain topics and terminology are bandied around in press releases, news articles, research, and funding announcements. These include Blockchain<\/a>, ICO<\/a>, IoT<\/a>, as well as AI<\/span><\/strong>, Machine Learning<\/span><\/strong> and Deep Learning<\/span><\/strong>. But what exists beneath the hype and how do we separate predictions from reality?\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n

The State of the AI report<\/a> is a yearly analysis by UK investors Nathan Benaich and Ian Hogarth which offers a 177 slides deep dive into all things AI based on invited contributions from well-known and up-and-coming companies and research groups. It\u2019s a massive document, rich in rabbit-hole worthy topics and definitely worth a read. We\u2019re taking a look at some of the key findings.<\/p>\r\n\r\n\r\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tTable Of Contents\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. Did the State of AI\u2019s predictions come true in 2019?\u00a0<\/a>
  2. Research\u00a0<\/a>
  3. What tools and resources are researchers using?\u00a0<\/a>
  4. A new generation of transformer language models are unlocking new NLP use-cases<\/a>
  5. Biology and Health sciences are repeating the benefits<\/a>
  6. The COVID Moonshot project\u00a0<\/a>
  7. Federated learning is booming<\/a>
  8. Improved validity in AI research<\/a>
  9. AI and Machine learning have gone mainstream\u00a0<\/a>
  10. Ethics on the front line\u00a0<\/a>
  11. 8 AI predictions for the next 12 months<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\r\n\r\n\r\n

    Did the State of AI\u2019s predictions come true in 2019?\u00a0<\/h2>\r\n\r\n\r\n\r\n
    \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
    2019 Predictions<\/strong><\/td>\r\nGrade<\/strong><\/td>\r\nEvidence<\/strong><\/td>\r\n<\/tr>\r\n
    New natural language processing<\/span> companies raise $100M in 12 months.<\/td>\r\nYes<\/td>\r\nGong.io ($200M), Chorus.ai ($45M), Ironscales ($23M), ComplyAdvantage ($50M), Rasa ($26M), HyperScience ($60M), ASAPP ($185M), Cresta ($21M), Eigen ($37M), K Health ($48M), Signal ($25M), and many more!<\/td>\r\n<\/tr>\r\n
    No autonomous driving company drives >15M miles in 2019.<\/td>\r\nYes<\/td>\r\nWaymo (1.45M miles), Cruise (831k miles), Baidu (108k miles).<\/td>\r\n<\/tr>\r\n
    Privacy-preserving ML adopted by a F2000 company other than GAFAM (Google<\/span>, Apple<\/span>, Facebook<\/span>, Amazon<\/span>, Microsoft<\/span>).<\/td>\r\nYes<\/td>\r\nMachine learning ledger orchestration for drug discovery (MELLODY) research consortium with large pharmaceutical companies and startups including Glaxosmithkline, Merck and Novartis.<\/td>\r\n<\/tr>\r\n
    Unis build de novo<\/em> undergrad AI degrees.<\/td>\r\nYes<\/td>\r\nCMU graduates first cohort of AI undergrads, Singapore\u2019s SUTD launches undergrad degree in design and AI, NYU launches data science<\/span> major, Abu Dhabi builds an AI university.\u00a0<\/td>\r\n<\/tr>\r\n
    Google has a major quantum breakthrough and 5 new startups focused on quantum ML are formed.<\/td>\r\nSort of<\/td>\r\nGoogle demonstrated quantum supremacy in October 2019! Many new quantum startups were launched in 2019 but only Cambridge Quantum, Rahko, Xanadu.ai, and QCWare are explicitly working on quantum ML.<\/td>\r\n<\/tr>\r\n
    Governance of AI becomes a key issue and one major AI company makes substantial governance model change.<\/td>\r\nNo<\/td>\r\nNope, business as usual.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/figure>\r\n\r\n\r\n\r\n

    Research\u00a0<\/h2>\r\n\r\n\r\n\r\n
    \r\n
    \"AI<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

    AI research is less open than you think: only 15% of papers publish their code<\/strong>.<\/p>\r\n\r\n\r\n\r\n

    The research notes that traditionally academic groups are more likely to publish their code than industry groups such as OpenA and DeepMind, who only published a limited amount. OpenAI and DeepMind.\u00a0<\/p>\r\n\r\n\r\n\r\n

    \u201cFor the biggest tech companies, their code is usually intertwined with proprietary scaling infrastructure that cannot be released. This points to a centralization of AI talent<\/strong> and computation as a huge problem.\u201d<\/p>\r\n\r\n\r\n\r\n

    What tools and resources are researchers using?\u00a0<\/h2>\r\n\r\n\r\n\r\n

    Benaich and Hogarth used conference papers to determine the most used framework<\/strong>. Of those that specify the frameworks used, 75% cite the use of PyTorch<\/strong> but not TensorFlow<\/a>. However, TensorFlow, Caffe and Caffe2 are still the workhorse for production AI. PyTorch offers greater flexibility and a dynamic computational graph that makes experimentation easier. JAX is a Google framework that is more math friendly and favoured for work outside of convolutional models and transformers.\u00a0<\/p>\r\n\r\n\r\n\r\n

    They note that traditionally performance is driven by bigger models, datasets and compute budgets<\/strong>, especially in big enterprises. For example, they estimate OpenAI\u2019s 175B parameter GPT-3 could have cost tens of millions to train. Experts suggest the likely budget was $10M.<\/p>\r\n\r\n\r\n\r\n

    However, Benaich and Hogarth also contend that for some use cases like dialogue, small, data-efficient models can trump large models. An example is PolyAI, a London-based conversational<\/a> AI company, who open-sourced their ConveRT model (a pre-trained contextual re-ranker based on transformers). Their model outperforms Google\u2019s BERT model<\/a> in conversational applications, especially in low data regimes, suggesting BERT is far from a silver bullet for all NLP tasks.<\/p>\r\n\r\n\r\n\r\n

    A new generation of transformer language models are unlocking new NLP use-cases<\/h2>\r\n\r\n\r\n\r\n

    GPT-3<\/a>, T5, BART are driving a drastic improvement in the performance of transformer models for text-to-text tasks like translation, summarization, text generation, text to code. For example, an unsupervised machine translation<\/span> model trained on GitHub<\/span> projects with 1,000 parallel functions can translate 90% of these functions from C++ to Java<\/span>, and 57% of Python<\/span> functions into C++ and successfully pass unit tests. No expert knowledge required, but no guarantees that the model didn\u2019t memorize the functions either.\u00a0<\/p>\r\n\r\n\r\n\r\n

    Biology and Health sciences are repeating the benefits<\/h2>\r\n\r\n\r\n\r\n
    \r\n
    \"AI<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

    Biology is experiencing its \u201cAI moment\u201d<\/strong> with over 21,000 papers in 2020 alone involving AI methods (e.g. deep learning, NLP, computer vision). Papers published since 2019 account for 25% of all output since 2000.<\/p>\r\n\r\n\r\n\r\n

    Deep learning on cellular microscopy accelerates biological discovery with drug screens.\u00a0AI-based system for mammography screening<\/a>\u00a0has also reduced false positives and false negatives in two large, clinically-representative datasets from the US and UK.<\/p>\r\n\r\n\r\n\r\n

    The COVID Moonshot project\u00a0<\/h2>\r\n\r\n\r\n\r\n

    Drug discoveries have gone open source<\/span> to tackle COVID-19<\/a>. Benaich and Hogarth assert this is a rare example of where AI is actively used on a clearly-defined problem that\u2019s part of the COVID-19<\/strong> response. An international team of scientists are working pro-bono, with no IP claims on a project called COVID Moonshot to crowdsource a COVID antiviral.<\/p>\r\n\r\n\r\n\r\n

    PostEra<\/a>\u2019s synthesis technology allowed the consortium to design \u2018recipes\u2019 for 2,000 molecules in under 48 hours. Human chemists would have taken 3-4 weeks to achieve the same task.\u00a0<\/p>\r\n\r\n\r\n\r\n

    Moonshot has received over 10,000 submissions from 365 contributors around the world, testing almost 900 compounds and identifying 3 lead series.\u00a0<\/p>\r\n\r\n\r\n\r\n

    Federated learning is booming<\/h2>\r\n\r\n\r\n\r\n

    Kicked off by Google in 2016, federated learning<\/strong> research has experienced almost 5x growth in the number of papers that mention federated learning from 2018 to 2019. More papers have been published in the first half of 2020 than in all of 2019.<\/p>\r\n\r\n\r\n\r\n

    OpenMined is an open-source community whose goal is to make the world more privacy-preserving by lowering the barrier-to-entry to private AI technologies. They extend major Machine Learning frameworks (PyTorch, Tensorflow, etc.) with techniques for privacy such as: Federated Learning, Homomorphic Encryption, Secure Multi-party Computation, and Differential Privacy. They also build integrations allowing for training across Cloud<\/span>, mobile OSs such as Android<\/span> and iOS, CPU<\/span>, GPU<\/span>, IoT, and Javascript (Web) technologies. This year they demonstrated the first open-source federated learning platform for web, mobile, server, and IoT.\u00a0<\/p>\r\n\r\n\r\n\r\n

    Improved validity in AI research<\/h2>\r\n\r\n\r\n\r\n

    A review of 20,000 recent AI-based medical imaging studies found that less than 1% of these studies had sufficiently high-quality design and reporting<\/strong>. Studies suffer from a lack of external validation by independent research groups, generalizability to new datasets, and dubious data quality. This year new international guidelines were drafted for clinical trial protocols <\/a>and reports <\/a>that involve AI systems in a bid to improve both quality and transparency. New requirements include:<\/p>\r\n\r\n\r\n\r\n