Margaret Maynard Reid is a machine learning engineer at Tiny Peppers. She currently works on research to showcase how AI can be used to create art. “I’m a machine learning GDE. I contribute to TensorFlow, write tutorials, and blog posts and speak at conferences. I am a community leader. I’m the lead organiser of GDE Seattle, and Seattle Data analytics, and Machine learning. This article offers an overview of some of the topics covered including a deep dive into GANs and their application in art and design, you’ll really want to take a look at the video below to watch the presentation and get the full experience, especially of the images!
Understand the terminology
We hear these words a lot, AI, machine learning, deep learning and computer vision. So what’s the difference?
- AI is the broadest playing that aims to create intelligent machines
- Machine learning is one way to achieve that intelligence.
- Deep Learning is a subset of machine learning, which has become quite popular in recent years.
We can use deep learning, machine learning, or non-machine learning algorithms to solve problems, such as computer vision. For natural language processing, computer vision is one way to interpret images with computers.
Computer vision overview
An overview of computer vision tasks:
- Image classification: Classify whether an image belongs to a particular one class
- Image classification (multi-label): Classify and image to multi classes
- Feature extraction: Extracting latent features of an image with CNN models.
- Object detection: Identify one or multiple objects within an image and their locations with bounding boxes.
- Segmentation: Classify whether each pixel of the image belong to a certain class
- Generative models (GANs): Two or more models trained simultaneously
Generative models to create art and design
Deep Dream (2015)
Margaret explains, “
Deep dream was an experiment that visualises the patterns learned by a neural network similar to when the child watches clouds tries to interpret random shapes, deep dream over, interprets and enhances the pattern to see in an image. It was really the first time we used a neural network to create some artistic art representation.
Image to Image Translation
In 2015 Margaret and her team were able to perform image style tests for by using convolution on your network feature extraction and apply the style of a style image and to a content image, and then produce a new image.
Generative Adversarial Networks
In 2014 Ian Goodfellow created generative adversarial networks. And since then, there are all kinds of variations of Gans. GANs are fun but difficult to train. Generative adversarial models have at least two network models which compete against each other, like a chess game. It’s a zero-sum game.
Deep convolutional GANs (DCGAN)
One of the earliest kinds of GANs. As Margaret explains, “The network uses random noise as input, then it has a generator that generates images and the discriminator critiques the degenerated images made by the generator, and as a result, teaches his generator how to become better at creating these fake images.
So we have random noises input and then the output are actually the images.”
Pix to Pix
Pix to Pix was introduced in 2016 pics and it was able to convert black and white images to colour images, sketches to high res photos, day to night, etc.
When you do training, you must have paired images which may not be readily available.
Cycle GAN (2017)
Similar to Pix to Pix without the need for paired training images. For example, Cycle GAN can convert between horses and zebras, winter and summer, and then a picture at regular photo converted to a Van Gogh painting.
Big GAN
In 2018, Google researchers experimented with Big GAN, “which is a large scale game that generates high fidelity class conditional images. One of the main challenges is again, that you don’t have a whole lot of control over the images that you’re generating with conditional GANS that I mentioned earlier, that you can condition based on a particular class or perhaps a text or image. ” However Style GAN was to change things:
Style GAN (2018)
StyleGAN is a novel generative adversarial network introduced by Nvidia researchers in December 2018. It allows for fine control of images. For example, you can create faces based on a particular pose, face shape, age, gender, hair colour, etc.
There are many other variations of GANs such as Image Inpainting which lets you edit images with a smart retouching brush. NVIDIA GPUs and deep learning algorithms make it possible to replace any portion of the image.
Image to text
Fashion++ (2019)
In 2019, Facebook AI came up with Fashion++, capable of recommending fashion changes. Not only is it generating images fashion, but it also may have some sort of sense of whether the changes in the image is fashionable or not.
Sketch2Code (2018)
In 2018, Microsoft AI lab created Sketch2Code. This is a web-based solution that uses AI to transform a handwritten user interface defined from a picture to a HTML markup code, which produces high-resolution design.
Datasets, Tools and Resources
As Margaret notes, “To train machine learning models you will need data” She has included in her presentation (and slides) extensive data resource lists including art and UI and UX design. Some of her examples include the Art Institute of Chicago which provides 5000 free high res images and art data on Kaggle.
According to Margaret, in terms of UI and UX design, data sets are a bit more limited. Rico mobile app datasets have UI layouts as well as segmentation to help with mobile application UI design. Kaggle’s 50 icon has 10,000 icons with 50 categories.
Project magenta: This is a project open-source research project by Google that uses machine learning to make music and art. You can find various project demos and code.
RunwayML: Runway ML is a community platform for creatives of all kinds to use machine learning tools in intuitive ways without any coding experience.
Artists and Machine Intelligence: Artists and machine learning intelligence is a programme that Google created to bring artists and engineers together to realise projects using machine intelligence.
Watch the rest of Margaret’s talk above for more resources and tools, including resources around ethics around AI design and designing human-centred AI products.