Free · sign in to read

One model. One GPU. Many people waiting.

This course is about what happens between a prompt and an answer, why serving a language model is slow and expensive by default, and the handful of ideas that make it fast and cheap. Written for engineers who have called a model API and never run one.

Start reading See the labs

6 parts · 28 modules · 20 labs · 28 modules written so far

What you will learn

Theory first, concepts only. The numbers come in the labs, measured on the path you choose.

Part 00

Before we start

The problem, then just enough of the machine to talk about it.

Part 01

One request on one GPU

What happens between a prompt and an answer, and what it costs.

Part 02

Many requests

The scheduler and the memory manager, where most of the throughput comes from.

Part 03

Fewer bytes, fewer steps

The techniques that change what one step costs.

Part 04

More than one GPU

Enough to read a production diagram.

Part 05

Running it

From an engine to a service, with the numbers to run it by.

Where the labs run

Pick one path. Every lab shows the commands and the expected output for the path you chose.

Kaggle

Free, no card

2 × T4, 16 GB each

30 GPU hours a week with a Kaggle account. The path most readers take. One lab is read-only here because the T4 has no FP8 KV cache.

Modal

Pay per second, card on file

1 × L4, 24 GB

Every lab runs in full, from a script, on an image that starts in seconds. About $0.80 an hour while a lab runs; the whole course is a few dollars.

Loading…