Skip to content
ML Academy

DL, NLP and LLMs

Interview question

  • seniorlevel
  • 15 minsuggested time
  • 2ML engineer, Applied AI

You serve a 7-billion-parameter decoder-only model in 16-bit precision on one 80 GiB GPU for a support assistant with 8,192-token contexts. Throughput is far below the benchmark, and requests queue. A teammate says "buy more GPUs". Using the hypothetical configuration of 32 layers, 8 key-value heads and head dimension 128, where is the memory going, and what are your options?

Tests: Estimating inference memory (weights and KV cache), connecting memory to concurrency, and knowing the levers (attention variant, cache quantisation, paging, batching, context limits).

day 42day 39day 54

Sign in to practise: answers are saved before model answers are shown.