S32 · paper simulation

Read a churn paper before believing its abstract

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
35 minabout
0stages committed

The brief

Your director forwards a preprint. The paper, its authors and its results are fictional, written for this exercise. Its abstract (paraphrased):

"ChurnPrompt fine-tunes a 7-billion-parameter instruction-tuned language model on customer records serialised as text. On a public telecom churn dataset, ChurnPrompt reaches ROC-AUC 0.91 versus 0.85 for XGBoost, a gain of 6 points, showing that language models are the new state of the art for tabular churn prediction."

Your company (simulated) is a telecom operator with 2 million subscribers and a monthly batch churn score used for retention offers.

Stage 1 of 5

Before reading the method section, you decide what you need to check.

Which checks belong in the reading protocol for this claim? Select all that apply.

How was the data split, and can the same customer appear in both training and test data?Were the baseline and new method tuned with equal effort, on data separate from the test set?Is there uncertainty across seeds or resamples, or is the 6-point gain from a single run?Could the public dataset have been in the language model's pretraining data?Is the model larger than the ones used in previous papers, since scale usually explains gains?