Build a sequence.
Anewidea…
- Read the prompt and text so far.
- Score possible next tokens.
- Select, append and repeat.
Each new token becomes part of the next prediction.
Shahzad Ali · Learning notes
Text, images & video.
How generative AI works across different media.
My visual notes on how models turn a prompt into text, an image or a moving scene, and the systems that make it possible.
Different media.
Different paths.
Models learn patterns during training, then use them to generate something new.
Refine a field.
→→
- Start with noise.
- Refine it using learned patterns and the prompt.
- Decode the result into pixels.
A simplified latent-diffusion example.
Carry it through time.
01
02
03
- Represent a sequence of frames.
- Refine across space and time.
- Generate movement and continuity.
Objects need to stay consistent as they move.
Text generation
Text builds a sequence
A language model processes the prompt and text generated so far, then scores possible next tokens. A decoding rule selects one. That token joins the context for the next prediction.
The transformer note follows the full text-generation loop, including embeddings and attention. The comparison here focuses on what changes between text, image and video generation.
The full token, embedding and attention explanationImage diffusion
An image emerges through refinement
A diffusion model learns to reverse a noise-adding process. During generation, it starts from noise and repeatedly refines a representation, guided by the prompt.
Latent diffusion performs this work in a compressed image space, then decodes it into pixels. The sampler and model determine the number of steps. Different image generators use different architectures and generation methods.
Generating with a trained modelVideo generation
Video adds time
A video model has to represent movement and continuity across a sequence: a person should remain recognisable as they walk, and objects should behave consistently from frame to frame.
OpenAI’s 2024 Sora report describes compressed visual data divided into spacetime patches and processed by a diffusion transformer. It also documents failures in physical interactions and long-range coherence. Audio generation depends on the specific model.
TransformersRetrieval-augmented generation · RAG
Retrieval brings extra material into the conversation
A retrieval system searches a collection for material relevant to a question. It supplies selected passages to the generator, giving the answer additional context. Embedding similarity is one way to find candidates.
For example: question → search company documents → retrieve relevant passages → add them to the prompt → generate an answer. At ordinary inference, this can change the available context while the trained weights stay fixed. Retrieval quality, document accuracy and answer checking still matter.
What context meansData centres and compute
The model needs physical machinery
Large AI systems run calculations on accelerators such as GPUs. Memory holds parameters and working data; networks connect machines; storage keeps datasets and checkpoints. Electricity and cooling support the whole system.
Training large models and serving many requests both create demand for computing capacity. Smaller models can run on a single computer. Capacity, energy use and infrastructure depend on the workload and deployment.
What a GPU does
Sources behind the technical details
Sources for these notes.
Reviewed . Technical details are linked to their sources. My perspective on how I use AI is personal.
- Primary research · First released
Language Models are Few-Shot Learners ↗
Brown and colleagues. Describes autoregressive language modelling, whole-model parameter counts and learning from context with fixed trained weights. Its results also document limitations.
- Primary research · First released
High-Resolution Image Synthesis with Latent Diffusion Models ↗
Rombach and colleagues. Describes diffusion in a learned compressed image representation, prompt conditioning and sampling trade-offs.
- Developer technical report · First released
Video generation models as world simulators ↗
OpenAI. Describes the 2024 Sora research system, spacetime patches and diffusion transformers, including observed limitations. This source concerns that visual research system.
- Primary research · First released
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗
Lewis and colleagues. Combines a retriever with a generator. The original study also trains components; retrieval during ordinary use can supply new context without updating generator parameters.
- Manufacturer announcement · First released
Blackwell-powered DGX SuperPOD ↗
NVIDIA. A concrete example of GPUs, memory, interconnects, storage and cooling in a large AI system. Product specifications and performance claims are the manufacturer’s account.