Learn inference engineering, from zero.

Your first model, measured

Run a small model on your laptop, call it over HTTP, and measure its speed.

What just happened: tokens, the loop, the two phases, and "will it fit?"

Explain how a model writes one token at a time, why the first token takes longest, and whether a model file fits in GPU memory.

Python for JavaScript developers (optional bridge)

Read and write the Python the labs use, starting from the JavaScript you already know.

PyTorch and Transformers for inference (no training)

Load a model with PyTorch and write your own decode loop on a laptop, with no training.

A model is a file of numbers

Open a model's files and explain what its weights, config and tokenizer are and how big they are.

Tokens in depth: tokenizers, templates, sampling, thinking and tool-call text

Predict token counts, format prompts correctly, control sampling, and find the thinking and tool-call text in a raw output.

Inside a transformer, for engineers

Follow one token through a transformer and point to where the weights and the KV cache come from.

Your first GPU, free: open it, check it, time it right

Open a free GPU, check its driver and versions, time its work correctly, and read its memory.

Will it fit? Weights and number formats

Work out how much GPU memory a model's weights need in any number format, and pick a GPU that fits.

The KV cache: memory that grows with every token

Size the KV cache for any model, context length and user count, and predict when memory runs out.

What a GPU is, for software developers

Read a GPU spec sheet and explain why memory bandwidth, not compute, limits most inference.

Why decode is slow: the roofline in plain words

Predict a model's fastest possible tokens per second on a GPU, and explain why batching raises throughput.

Linux, Docker and Kubernetes in one weekend

Run, inspect, break and fix an inference-shaped service in Docker and on a local Kubernetes cluster, with no GPU.

vLLM on a free GPU without pain or a surprise bill

Install vLLM on a free GPU within an hour, with pinned versions and a shutdown rule, and fix the setup errors you meet.

Your first vLLM server

Serve a model with vLLM's OpenAI-style API and read its startup log and main settings.

Load testing that tells the truth

Run a load test that is repeatable, measure its noise, and read a real latency and throughput curve.

Queues, overload and SLOs

Predict when a server falls over, read the signs of an overload incident, and protect the server with limits and SLOs.

Batching: from one request to hundreds

Build a continuous batcher with tests, and measure what batching gives one user and what it gives everyone.

Paged KV memory and preemption

Build a KV block allocator with preemption, and tune an engine's memory so requests stop getting preempted.

Scheduling: chunked prefill, CPU overhead, CUDA graphs, the TTFT-ITL trade

Stop long prompts from freezing other users, and explain where CPU overhead, CUDA graphs and compilation fit.

Prefix caching: reuse instead of recompute

Make repeated prompt prefixes nearly free, design prompts that hit the cache, and measure the hit rate.

Read a real engine, then choose one

Trace a decode step through a small engine's code, then choose a production engine with evidence.

Quantization: smaller models, same answers?

Serve a quantized model, measure its gains, and prove that its answers hold up.

Speculative decoding: faster when it works, slower when not

Turn on speculative decoding, measure how often the draft is accepted, and find where it stops paying.

Structured output and multi-LoRA serving

Return JSON and tool calls that always parse, and serve many adapters on one base model.

Serving reasoning and tool-calling models

Serve reasoning and tool-calling models so the right fields come back, cap thinking cost, and prove tool-call accuracy with a fixed test set.

Go for platform engineers (optional bridge)

Read and write enough Go to build a gateway or a Kubernetes controller in it.

The API layer: streaming, cancellation, limits, auth, TLS

Build a production API in front of an engine, with streaming, cancellation, rate limits, keys and TLS.

Containers, Kubernetes and operators for GPU inference

Run an engine on Kubernetes with GPUs, honest health checks and safe rollouts, and write a small operator.

Routing and load balancing for LLMs

Route requests so caches stay warm and no replica melts, behind a gateway that can fail over between providers.

Autoscaling, cold starts and the OS under the server

Scale on the right signal, cut cold starts from minutes to seconds, and explain the operating-system work inside a cold start.

Observability: see performance, quality and hardware

Build dashboards and alerts that catch latency, capacity, quality and GPU problems early.

Correctness, reliability and incidents

Keep answers right and the service up with deploy gates and load shedding, and work real incidents from the first alert to the postmortem.

The serving plan: requirements, capacity and cost

Turn a vague request into a serving plan with SLOs, a model choice, a fleet size and a cost per million tokens.

Fleet and capacity: regions, quotas and failing GPUs

Keep enough healthy GPUs in the right regions, and pull a failing node out of service safely.

Multi-GPU basics: tensor parallelism

Split a model across GPUs and predict when more GPUs stop helping.

Big models: pipeline and expert parallelism, MoE

Explain how the largest mixture-of-experts models run across many GPUs, and read a real production design.

Disaggregation, KV tiers and long context

Decide when to split prefill from decode, and when to move the KV cache to CPU memory or disk.

Profiling: find where the time goes

Profile a request from Python down to GPU kernels and name the bottleneck with evidence.

Numerics and determinism

Explain why one prompt can give different outputs, and test numerics across batch sizes and versions.

GPU programming for inference engineers

Write and time your first GPU kernels, and explain how threads, warps and memory set their speed.

Attention kernels and backends

Explain FlashAttention and paged-attention kernels, and fix unsupported-backend errors.

Speech: speech-to-text and streaming text-to-speech

Serve speech-to-text and text-to-speech models to real-time targets, and measure the time to first audio.

Vision, image generation and embeddings

Serve vision-language, image generation and embedding models, and explain how their bottlenecks differ from chat.

The proof of concept: from a customer brief to a memo

Turn a customer's brief into a working proof of concept, a benchmark and a one-page memo.

The job market and your track

Read a job post's real level and skills, and name the track and the portfolio you will build.

Proof of work: open source and write-ups

Publish work a hiring manager can check in minutes, including a pull request a maintainer reviewed.

Interview practice

Practice system-design, coding and debugging interviews under time limits, and finish one take-home.

Capstone: ship a production inference service

Ship a production inference service for one real use case, proven by numbers, a live drill and a design review.