Types of Generative Models

You can use generative AI to accomplish many tasks, but some generative models are better than others at specialized tasks. For example, if you want to create text that looks like it could be written by a human, you could choose a transformer model like Gemini or ChatGPT. If you wanted to create an original image based on a prompt you describe, you might use a diffusion model like Stable Diffusion or DALL-E. You might not have the best results if you tried to ask a diffusion model to write you a paragraph or a transformer model to paint you a picture. 

Learn more about the generative models you can use to create text, images, code, synthetic data, and things like generative adversarial networks, variational autoencoders, diffusion models, autoregressive models, and recurrent neural networks. 

What are generative AI models?

Generative AI models can create original pieces of data that look similar to but are fundamentally different from the material they saw during training. Developers and AI researchers use a large amount of data to train these algorithms and help them understand how data relates to one another. These models can analyze the patterns underlying datasets and use that information to generate similar data. The key point that makes these models different from one another is the strategy they use to understand and model data before applying what they learned to create a new item. 

Types of generative models

You can use many different kinds of generative models to create images, text, videos, animations, code, synthetic data, and more. A few of the more common types of AI generative models include generative adversarial networks (GANS), variational autoencoders (VAEs), diffusion models, recurrent neural networks (RNNs), and autoregression models. Within each of these categories, you can find even more specific types of generative models that use different strategies to accomplish the goal of generating content. Explore how GANs, VAEs, RNNs, diffusion models, and autoregression models work and what you can do with them. 

Generative adversarial networks (GANs)

Generative adversarial networks are neural networks that generate data through a competitive game between two neural networks. The first network, the generator, tries to win the game by creating a fake output that looks like the model’s training material. The second network, the discriminator, tries to win the game by looking at patterns it has learned about the training material and spotting the fake output from the generator. The generator suggests an output, and the discriminator marks it as a fake. Both networks go back and forth using machine learning to improve as they iterate until the generator can ultimately create an output that the discriminator can’t tell is fake. 

GAN applications: GANs are good tools for creating new images, building 3D models from 2D images, extrapolating data from incomplete data sets, and generating training data for other AI models. You can also use a GAN to identify false images or forgeries, or create data or malware samples to help you test security systems or antivirus software. 

Variational autoencoders (VAEs)

A variational autoencoder is a generative AI model that also uses two parts. The first part, the encoder, reduces the input to latent variables, the key variables that explain how other variables are distributed. Essentially, the encoder simplifies the input down to bare parts with only the most necessary information, which, in some cases, is difficult for humans to observe in the data. The decoder's second part of the model rebuilds the compressed input into the eventual output, which is a unique item based on the input and similar items from the model’s training material. 

All autoencoder models have these two parts, but VAEs are specifically good at generating unique content because of the way they encode latent variables. VAEs create a probabilistic representation of latent space, which means that they aren’t mapping the exact latent variables but rather estimating the probability of variables, allowing for variation and a unique output without losing the key characteristics of the original data. 

Variational autoencoder applications: VAEs can be used to create or reconstruct images and detect fraudulent activity by understanding the underlying variables that mark such activity, and in scientific research applications, among other uses. For example, a 2023 paper explored the potential benefits and challenges of using VAE models to generate chemical compounds to speed up research [1].

Recurrent neural networks (RNNs)

Recurrent neural networks are models that allow data to move through feedback loops, allowing the model to remember data that passed through previously when analyzing the data passing through currently. A neural network is a recurrent network that contains layers of interconnected nodes that make small calculations and manipulate the data at each layer. Other types of neural networks move the data from input to output without looping back, which is the distinguishing characteristic of an RNN. This mechanism makes RNNs a very good tool for understanding data that relies on a sequence or time. 

Recurrent neural network applications: You can use an RNN to analyze time-series data, such as the performance of an investment or changes that occur over a period of time. You can also use RNNs for sequential data, such as putting words into sentences. Since word order matters for sentences to have meaning, an RNN’s ability to understand sequential data makes it a tool you can use for natural language processing. 

Diffusion models

A diffusion model is a generative AI that creates new work by taking input and adding noise to distort and destroy the image. During training, a diffusion model studies the effects of adding noise and how it destroys data. When generating new work, the diffusion model uses a reverse diffusion process, starting with a sample of pure noise and restoring the data until it resembles something the model has seen in the training material. 

Diffusion applications: The most common reason you might use a diffusion model is to generate new images. You can also use them to create videos and do medical research.

Autoregressive models

Autoregressive models can generate text or images by predicting what component of data could follow the components of data it already knows. For example, when generating text, the model will predict the words most likely to go in each position by understanding how words relate to one another within a vast library of training material. A particularly interesting type of autoregressive model is the transformer model, which uses self-attention to understand not only the patterns of how data relate to one another but also which parts of the data are more important to pay attention to within the context of the prompt. These models form the technology for generative AI tools you may be familiar with, like ChatGPT. (“GPT” stands for generative pre-trained transformer.)

Autoregressive applications: You can use autoregressive models for time series analysis and forecasting because of their ability to predict what will come next in a sequence of data points. You can also use autoregressive models for large language models to perform natural language processing and generate text. 

Who uses generative models?

Many careers that work to help create and improve generative AI models have similar job responsibilities with different job titles. A few job titles that you might consider in this field include AI or machine learning engineer, AI researcher, or AI developer. 

AI/ML engineer

Average annual base salary in the US (Glassdoor): $122,823 [2]

Job outlook (projected growth from 2023 to 2033): 26 percent [3]

As an AI/ML engineer, you will work on a team to develop new AI applications. You will apply AI and machine learning principles to help people increase efficiency, cut costs, and make more informed decisions based on data. In this position, you may also work to maintain or improve AI or ML applications after development. As an engineer, you will focus on projects like AI infrastructure, systems design, and data structures. 

AI researcher

Average annual base salary in the US (Glassdoor): $99,352 [4]

Job outlook (projected growth from 2023 to 2033): 26 percent [3]

AI researchers and AI engineers are titles often used to describe similar careers using AI principles to solve real-world problems. In this role, you will help create applications using AI, designing, building, training, and developing AI solutions. As an AI researcher, you may work to create new algorithms that represent breakthroughs in AI technology, or you may work in other fields using AI algorithms to advance research, such as biotechnology. 

AI developers

Average annual base salary in the US (Glassdoor): $110,972 [5]

Job outlook (projected growth from 2023 to 2033): 17 percent [6]

As an AI developer, you will work with a team to develop AI applications. The difference between an AI developer and an AI engineer is the type of projects you’ll work on: As a developer, you’re more likely to work on projects for an end user than projects that contribute to AI application architecture. In this role, you’ll work with a team to develop, train, and test AI applications. 

Continue exploring machine learning on Coursera

Learning the different types of generative AI can help you understand which models will work best for your projects. If you want to learn more about AI and machine learning techniques, consider taking a course online to learn new skills and demonstrate your credentials to future employers. Depending on your career goals, you can find programs like the IBM AI Developer Professional Certificate or the AI For Business Specialization offered by the University of Pennsylvania on Coursera.