In brief: A couple of years ago, Nvidia introduced an AI tool that could take a rough doodle and produce a photorealistic image in real-time. It used generative adversarial networks dubbed "GauGAN," a nod to post-impressionist artist Paul Gauguin. Now just over two years later, Nvidia finally has the app ready for public use.
On Wednesday, Nvidia announced a new app called Canvas. It's still in beta but is available for anyone to use. It's so cool we already have a safe download link ready for those who want to try it out. Keep in mind that you will need Windows 10 and an Nvidia GeForce Quadro or Titan RTX GPU with driver 445 or later (download 471.11 here).
Canvas is the same GauGAN app introduced two years ago, but developers have vastly improved its interface to be tighter and more user-friendly (video: new above, old below). It has also benefitted from two more years of AI training on millions of images.
Generative adversarial networks (GANs) use two neural networks---a generator and a discriminator. The generator, as the name suggests, generates something, in this case, an image. It sends the created content to the discriminator, which evaluates it based on training from millions of real-world samples. It then provides feedback to the generator on how to improve the content---in this case, pixel-by-pixel.
"This technology is not just stitching together pieces of other images, or cutting and pasting textures," said Nvidia's Vice President of Applied Deep LearningResearch Bryan Catanzaro said in 2019. "It's actually synthesizing new images, very similar to how an artist would draw something."
The result is like having a magic paintbrush that automatically layers in textures as you paint crude segmentation maps. Artists can work on the segmentation map or on the generated image itself. Either way, the end result is a very convincing photo (below) that can be further manipulated with Photoshop or other tools.
The palette has several types of textures, from trees and grass to water and clouds. The cool thing (other than just painting trees and rocks with one stroke) is that the textures are smart. Water will automatically produce appropriate reflections. If you have a scene with grass and trees, you can quickly change the grass to snow, and the trees will become barren, creating a wintery look.
The Canvas app focuses on creating natural landscapes. However, Nvidia says that the underlying networks can generate other landscape features such as people, roads, and buildings.
If you are into the technical aspects of the GauGAN AI, check out Nvidia's academic paper, which it presented at the 2019 Computer Vision and Pattern Recognition Conference.