Practice

35 questions

• Statistics

  Leadership is tired of "not significant" verdicts and asks you to redesign how experiment results are reported so they can answer "how likely is it that this change makes money, and how much could we…
  staff20 min

• Analytics

  You join an airline's commercial analytics group. There are 400 dashboards, conflicting KPI definitions, and leadership says "we have lots of data but no answers".
  staff25 min

• Classical ML

  A bank's small-business credit team uses a logistic-regression scorecard. A new gradient-boosting model improves ROC-AUC from 0.74 to 0.79 on a holdout.
  staff25 min

• Forecasting

  In the academy's synthetic airline data, a competitor increases capacity by 25% on three routes from 2024-03-01.
  staff25 min

• DL, NLP and LLMs

  Your company is launching LLM-generated summaries of customer-service conversations for supervisors across four languages.
  staff25 min

• RAG and agents

  Five product teams are building RAG features on a shared retrieval platform. Documents change daily, teams change prompts weekly, and the model provider updates models a few times a year.
  staff30 min

• Experimentation and causality

  A team wants to A/B test a new dynamic pricing algorithm by randomising users: half see fares from the new algorithm, half from the old one, on the same flights.
  staff25 min

• Production and system design

  Design the product recommendation system for an e-commerce site with 20 million monthly users and 2 million products. Recommendations appear on the home page and product pages.
  staff45 min

• Production and system design

  Design a real-time card-payment fraud detection system for a bank processing a few thousand transactions per second at peak. Decisions must be made inside the authorisation flow.
  staff45 min

• Production and system design

  Design the ranking system for flight search results on a travel site: a user searches a route and date, and you must order hundreds of itineraries.
  staff45 min

• Production and system design

  Your company's LLM features (support assistant, document summarisation, internal search) now cost more than planned, and the CFO wants a 40% reduction in six months without hurting quality.
  staff30 min

• Business judgment

  You lead a team of six. Leadership hands you eight AI/ML project requests for next year (a GenAI customer assistant, demand forecasting upgrade, fraud model refresh, dashboard automation, and four…
  staff25 min

• Business judgment

  Over a year, your team's recommendation model improved click-through rate by 15% in successive A/B tests, but revenue per visitor is flat and returns increased.
  staff20 min

• Communication

  Tell me about a time you set technical direction across several teams (for example, a shared evaluation standard, a feature platform, or metric definitions) where nobody reported to you.
  staff15 min

• Classical ML

  A lender's credit-decision model approves 30,000 of 100,000 applicants (hypothetical) and observes repayment only for approved loans, where 4.0% (1,200) default.
  staff25 min

• Forecasting

  Three teams forecast the same quantity at different levels: network planning forecasts total weekly demand, route managers forecast their own routes, and finance consumes a budget number.
  staff25 min

• Experimentation and causality

  Your company runs about 200 A/B tests a year. The product organisation reports that 25 'winning' tests shipped last year, with measured lifts that sum to +25% in the key metric.
  staff25 min

• Business judgment

  Nine months into a demand-forecasting rebuild, the team has reached 70% of the accuracy-improvement target. The sponsor says "we have already spent 1.2 million; we must finish".
  staff25 min

• Communication

  The executive team must decide this week whether to launch an AI-assisted pricing feature to all markets.
  staff20 min

• DL, NLP and LLMs

  A customer-support routing step calls a large hosted LLM 30 million times a month. An engineer proposes distilling it into a small self-hosted model.
  staff25 min

• RAG and agents

  A legal-operations team asks for an assistant that answers questions over a set of about 300-page contract bundles. A vendor demo simply pastes the whole bundle into a long-context model.
  staff25 min

• Production and system design

  A churn model scores 20,000 customers a week. Churn is defined as no activity for 60 days, so the true outcome for a scored customer is known only two months later.
  staff25 min

• Production and system design

  Leadership wants ML-backed services to have service-level objectives like the rest of engineering, but a model can be fast and available while giving worse answers.
  staff25 min

• Forecasting

  The demand-planning team asks for two months of work to take weekly SKU-store forecast error (WAPE) from 14% to 12%. You are asked to decide whether it is worth it.
  staff25 min

• Experimentation and causality

  Over a year your product organisation ran 300 experiments and shipped the 25 "wins" (significant at 0.05, positive).
  staff25 minin review

• Analytics

  Marketing adds up the bookings that each channel's own platform report claims: paid search 45,000, social 30,000, email 25,000, affiliate 20,000 (hypothetical monthly figures;
  staff25 min

• Business judgment

  Your team of five engineers (about 100 productive engineer-days per month) operates 14 production models, each costing around 4 engineer-days a month in monitoring, retraining and incident work, so…
  staff25 min

• Communication

  Tell me about a time you decided to stop or significantly de-scope a project that had a lot of sunk cost, and how you brought the people involved with you. (Behavioural: answer using STAR.)
  staff15 min

• Production and system design

  Six teams each own several models, 40 in total, retrained weekly on different stacks. Incidents keep happening because a retrained model is promoted with a regression nobody noticed for days.
  staff40 min

• Classical ML

  A lender's credit model is retrained yearly on loans it approved. Leadership now wants to raise the approval rate from 40% to 60% of applicants and asks you to certify that the new model is safe.
  staff25 minin review

• DL, NLP and LLMs

  Whiteboard case. An airline's contact centre wants one intent-classification and summarisation capability for customer messages in Turkish, English and a long tail of other languages, replacing…
  staff30 min

• Forecasting

  Whiteboard case. A retailer wants one forecasting platform for 50,000 SKU-store series (daily, 28 days ahead) that feeds replenishment, with category managers entitled to override numbers.
  staff40 minin review

• Production and system design

  Whiteboard case. Design a system that chooses which of six ancillary offers (seat, bag, lounge, meal, upgrade, none) to show each customer during booking, 200,000 impressions per day.
  staff40 min

• Business judgment

  The CFO asks you to report the annual business value created by the company's ML systems. Five teams each submit a number from their own A/B tests: 6.0, 5.2, 4.1, 3.0 and 2.2 million (hypothetical).
  staff25 minin review

• Communication

  Operations has decided to redefine the company's on-time performance (OTP) KPI: from "share of operated flights arriving within 15 minutes of schedule, measured at wheels-on" to "share of scheduled…
  staff20 min