LLMs, GenAI, RAG, agents, evaluation · insight-0572
Pick a generative model family by the constraint that binds, not by popularity
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- retail, e-commerce
Summary: situation, action, reason, boundary
Situation
A retailer wants synthetic product images for layout testing and assumes the newest diffusion model is the only option.
Action
List requirements (fidelity, diversity, sampling speed, likelihood or latent access, training stability, data size) and match them to families, autoregressive, VAE, GAN, diffusion or latent diffusion, then test the top two on your data.
Reason
Families optimise different objectives. VAEs maximise a likelihood bound with an explicit latent space, GANs train adversarially and sample in one pass, and diffusion models denoise iteratively, with latent diffusion reducing compute.
Boundary
Image, audio, tabular and sequence generation decisions.
Symptom or risk
The chosen family misses the real constraint, such as generation speed, a usable latent space or likelihood estimates, and costs more than needed.
Exceptions
Text generation is dominated by autoregressive language models; this choice matters mostly for other modalities.
Trade-offs
Each family trades fidelity, diversity, speed and training difficulty differently.
Pitfalls
Comparing families on cherry-picked samples instead of a defined metric.
Lessons and concepts
Claims and their sources
Kingma and Welling introduce the variational autoencoder, trained by maximising a variational lower bound with a learned latent representation.
Goodfellow et al. train a generator against a discriminator in a minimax game.
Ho et al. generate samples by learning to reverse a gradual noising process, and Rombach et al. run diffusion in an autoencoder's latent space to cut compute.