Skip to content
ML Academy

DL, NLP and LLMs

Interview question

  • seniorlevel
  • 15 minsuggested time
  • 3Applied AI, Data scientist, ML engineer

You compare two prompts for an LLM that extracts the refund reason from customer messages, on the same 200 labelled messages. Prompt A gets 156 right (78%), prompt B gets 164 right (82%). Of the 200, 14 were right with A only and 22 right with B only (hypothetical counts). The team wants to ship B and say "B is 4 points better". The prompt had been picked as the best of about ten variants tried on these 200. What do you tell them, and what do you do next?

Tests: Comparing two systems on the same evaluation items with a paired test on discordant pairs, interpreting a small difference with an interval, sizing the evaluation set, and accounting for selection among many variants.

day 45day 4day 47

Sign in to practise: answers are saved before model answers are shown.