Books, papers and documentation

Pavel Dmitriev, Somit Gupta, Dong Woo Kim, Garnet Vaz · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017), pp. 1427-1436 · 2017

Checked 2026-10-11: Read the full text of the Microsoft Research PDF. Section 4: sample ratio mismatch detected with a chi-square test, usually implies severe selection bias. 5.2 ratio metrics can mislead when the denominator changes; report numerator and denominator counts. 5.4 underpowered metrics; 5.5 borderline p-values; 5.6 continuous monitoring and early stopping (or extending) increases Type I error; 5.8 segments affected by treatment produce Simpson's paradox, test SRM per segment, recursive segmenting is multiple testing, use corrections such as Bonferroni; 5.9 outliers, capping or log transforms; 5.10 novelty and primacy effects, segment effect by day or visit. DOI 10.1145/3097983.3098024 confirmed via Semantic Scholar.

Ivan P. Fellegi, Alan B. Sunter · Journal of the American Statistical Association 64(328), pp. 1183-1210 · 1969

Checked 2026-10-12: Crossref metadata (2026-10-12): title, two authors, JASA vol. 64 issue 328, pp. 1183-1210, 1969. Taylor & Francis page returned 403; content statement (a decision model over comparison of records with three outcomes link / non-link / possible link, optimal procedure being a likelihood-ratio rule under conditional independence of comparison fields) read from secondary descriptions of the abstract and a university-hosted copy listing, not from the publisher page.

David Colquhoun · Royal Society Open Science 1(3), 140216 · 2014

Checked 2026-10-12: PubMed Central record via search results: title, author, R. Soc. open sci. 1(3) 140216, 2014, DOI 10.1098/rsos.140216. Abstract: a p-value of 0.05 used to claim a discovery gives a wrong conclusion at least 30% of the time, more when experiments are underpowered; a 2015 commentary notes the result depends on an assumed prior probability of a real effect. Abstract-level reading only.

Alex Deng, Ulf Knoblich, Jiannan Lu · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2018); arXiv:1803.06336 · 2018

Checked 2026-10-11: arXiv page (title, authors, v4 12 Sep 2018) and full-text PDF section 3.1: distinguishes randomization unit from analysis unit; when randomizing by user but computing page-level metrics the observations are not i.i.d., and the metric is written as a ratio of two averages of randomization-unit quantities so the delta method gives its variance. KDD 2018 venue and DOI 10.1145/3219819.3219919 from the Semantic Scholar record.

Andrew Gelman, John Carlin · Perspectives on Psychological Science 9(6), 641-651 · 2014

Checked 2026-10-11: Read the author PDF (sites.stat.columbia.edu/gelman/research/published/retropower_final.pdf): title, authors, journal 9(6) 641-651, 2014. Defines Type S (wrong sign) and Type M (exaggeration ratio) errors; in noisy small-sample settings significant results can be misleading, and when power falls well below 0.5 the exaggeration ratio grows large.

Anastasios N. Angelopoulos, Stephen Bates · Foundations and Trends in Machine Learning 16(4), 494-591 · 2023

Journal version of the arXiv tutorial (2107.07511); prefer this for formal citation.

Checked 2026-10-11: Title, authors, FnT ML 16(4) pp. 494-591, March 2023 in Crossref DOI metadata.

Yoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society, Series B (Methodological) 57(1), 289-300 · 1995

Checked 2026-10-11: Oxford Academic landing page: title, authors, JRSS-B 57(1) pp. 289-300, 1995; summary introduces controlling the expected proportion of falsely rejected hypotheses (false discovery rate) and proves a sequential Bonferroni-type procedure controls it for independent test statistics.

Aleksander Fabijan, Jayant Gupchup, Somit Gupta, Jeff Omhover, Wen Qin, Lukas Vermeer, Pavel Dmitriev · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2019) · 2019

Checked 2026-10-11: Semantic Scholar record for the DOI (ACM page returned 403): title, seven authors, KDD 2019 and abstract. Abstract defines SRM as an observed sample ratio that differs from the expected one, calls it a symptom of many data-quality issues, says ignoring it can make a bad change look good, and describes a taxonomy from experiments at four companies.

Alex Deng, Victor Hu · Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM 2015) · 2015

Checked 2026-10-12: Microsoft ExP page via search results: title, two authors, WSDM 2015. Content: when a feature reaches few users, analysing all users dilutes the effect and power; practitioners analyse triggered users and then dilute, and these formulas are often applied incorrectly, especially for ratio metrics; the paper merges the steps. Abstract-level reading only.

W. S. Robinson · American Sociological Review 15(3), 351-357 · 1950

Checked 2026-10-12: Citation and content from several secondary accounts and a 2009 reassessment listing (JSTOR stable 2087176; paper not opened directly). Content: ecological (area-level) and individual-level correlations can differ sharply; the 1930 census state-level foreign-born versus literacy correlation had a different sign from the individual-level one.

Daniël Lakens · Social Psychological and Personality Science 8(4), 355-362 · 2017

Checked 2026-10-11: PubMed Central full text (PMC5502906): title, author, journal 8(4) 355-362, 2017. TOST tests against lower and upper bounds set by the smallest effect size of interest; a non-significant result cannot show an effect is absent, equivalence testing can reject effects large enough to matter.

Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn · Psychological Science 22(11), 1359-1366 · 2011

Checked 2026-10-12: SAGE and SSRN records via search results: title, authors, Psychol Sci 22(11) 1359-1366. Content: flexibility in choosing dependent variables, sample size, covariates and reported conditions substantially raises the false-positive rate above the nominal 5%, shown by simulation and two experiments. Abstract-level reading only.

David J. Spiegelhalter · Statistics in Medicine 24(8), 1185-1202 · 2005

Funnel plots with control limits by unit size instead of league tables; Project 005.

Checked 2026-10-11: Title, author, Statistics in Medicine 24(8) pp. 1185-1202 in Crossref DOI metadata (Crossref issued date 2004, online-first; volume 24 is the 2005 volume).

F. J. Anscombe · The American Statistician 27(1), pp. 17-21 · 1973

Checked 2026-10-12: Crossref metadata (2026-10-12): title, author, journal, vol. 27 issue 1, pp. 17-21, February 1973. Publisher page returned 403, so the statement that four small datasets share the same usual summary statistics yet look very different is confirmed only through secondary course pages and the Matejka and Fitzmaurice paper, which cites it as the precedent.

Gerd Gigerenzer, Ulrich Hoffrage · Psychological Review 102(4), 684-704 · 1995

Day 3 natural-frequency presentation of Bayes' theorem and base rates.

Checked 2026-10-11: Opened the authors' manuscript PDF (pages.ucsd.edu copy): first page prints Psychological Review 102(4), 1995, 684-704, title and both authors; abstract states Bayesian computations are simpler in frequency formats and that naive participants used Bayesian algorithms for up to 50% of inferences with frequency formats. DOI resolves to doi.apa.org.

Beata Strack, Jonathan P. DeShazo, Chris Gennings, Juan L. Olmo, Sebastian Ventura, Krzysztof J. Cios, John N. Clore · BioMed Research International 2014, Article 781670 · 2014

Dataset paper and attribute dictionary for Diabetes 130-US Hospitals; Project 005.

Checked 2026-10-11: Title, authors, BioMed Research International 2014 in Crossref DOI metadata; the UCI page points to its Table 1 for attribute definitions.

Alex Deng, Ya Xu, Ron Kohavi, Toby Walker · Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM 2013) · 2013

CUPED variance reduction with a pre-period covariate (Day 4 power, Day 5 experiments).

Checked 2026-10-11: Microsoft ExP paper page (exp-platform.com/cuped): title, four authors, WSDM 2013 Rome; summary says CUPED uses pre-experiment data to reduce metric variability and reports roughly 50% variance reduction on Bing, equivalent to the same power with half the users or half the duration. DOI 10.1145/2433396.2433413 and authors confirmed in the Semantic Scholar record (ACM page returned 403).

Tilmann Gneiting · Journal of the American Statistical Association 106(494), 746-762 · 2011

Checked 2026-10-12: Crossref metadata and the arXiv abstract page 0912.0902: comparing forecasts by average error scores can mislead unless the scoring function matches the task; a score is consistent for the mean iff Bregman, and quantiles require generalised piecewise linear scores; expectations and quantiles are elicitable.

Victor Chernozhukov, Ivan Fernandez-Val, Alfred Galichon · Econometrica 78(3), 1093-1125 · 2010

Checked 2026-10-12: Crossref metadata (title, Econometrica 78(3) pp. 1093-1125, 2010; Crossref lists no authors) and the arXiv abstract page 0704.3649 (authors Chernozhukov, Fernandez-Val, Galichon): monotone rearrangement of non-monotone estimated quantile curves, argued to be closer to the true curve in finite samples.

Jennifer A. Hutcheon, Arnaud Chiolero, James A. Hanley · BMJ 340, c2289 · 2010

Checked 2026-10-12: Citation confirmed from BMJ link and citing papers (search results; BMJ page itself not opened). Content: random error in an exposure pulls estimated regression slopes toward the null, error in an outcome mainly widens intervals; a larger sample does not remove the exposure-error bias.

Jeffrey S. Vitter · ACM Transactions on Mathematical Software 11(1), pp. 37-57 · 1985

Checked 2026-10-12: Crossref metadata (2026-10-12): title, author, TOMS vol. 11 issue 1, pp. 37-57, 1985. ACM page not opened (DOI redirects to dl.acm.org). Reservoir sampling definition (k items uniformly at random from a stream of unknown length, one pass, fixed-size memory) confirmed on the NIST Dictionary of Algorithms and Data Structures entry, which cites this paper.

Adrian G. Barnett, Jolieke C. van der Pols, Annette J. Dobson · International Journal of Epidemiology 34(1), 215-220 · 2005

Checked 2026-10-11: Oxford Academic page: title, authors, IJE 34(1) 215-220, Feb 2005 (online Aug 2004). Abstract: natural variation in repeated data can look like real change; the effect is larger with more measurement error and when follow-up is limited to a subgroup selected for extreme baseline values; reduce it by design and suitable analysis.

Leo Breiman · Statistical Science 16(3), from p. 199 · 2001

Classic essay contrasting data modeling with algorithmic modeling; use it to frame prediction versus inference.

Checked 2026-10-11: Title, author, Statistical Science vol. 16 issue 3, first page 199, 2001/08 and DOI in citation metadata on the Project Euclid landing page.

Xiao-Li Meng · Annals of Applied Statistics 12(2), 685-726 · 2018

Checked 2026-10-12: Project Euclid record and search summaries (full text paywalled; a Harvard-hosted PDF exists): title, author, AoAS 12(2) 685-726, DOI 10.1214/18-AOAS1161SF. The paper decomposes the error of a sample average into data quality (correlation between outcome and the recording/response indicator), data quantity and problem difficulty, and shows that without probability sampling the error relative to the 1/sqrt(n) benchmark grows with the population size. Abstract-level reading only.

Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman, Douglas G. Altman · European Journal of Epidemiology 31, 337-350 · 2016

Day 4 confidence intervals, p-values, non-significance, power; Day 5 significance vs importance.

Checked 2026-10-11: Opened the PubMed Central full text (PMC4877414): title, seven authors, Eur J Epidemiol 31:337-350 (2016). The numbered list of 25 misinterpretations includes #1 (P value as probability the test hypothesis is true), #6 (P > 0.05 means no effect was observed), #7 (significance means an important relation), #19 (a specific 95% CI has a 95% chance of containing the true effect).

Andrew Gelman · Statistical Science 22(2), 153-164 · 2007

Checked 2026-10-12: arXiv abstract page (0710.5005, submitted 26 Oct 2007): title, author, Statistical Science 22(2) 153-164. Abstract: models of survey responses should account for the variables that drive inclusion, nonresponse, weighting and clustering; this can require thousands of poststratification cells and general multilevel models are hard to build. Abstract-level reading only.

Ronald L. Wasserstein, Nicole A. Lazar · The American Statistician 70(2), 129-133 · 2016

The ASA's six principles on what p-values do and do not mean; cite it for correct interpretation of significance tests.

Checked 2026-10-11: Title, authors, The American Statistician vol. 70 issue 2, pp. 129-133, published online 09 Jun 2016, free access, on the Taylor & Francis landing page (opened in a browser; plain HTTP fetch was blocked).

Yoav Benjamini, Daniel Yekutieli · Annals of Statistics 29(4), 1165-1188 · 2001

Checked 2026-10-12: Project Euclid abstract via search results: title, authors, Ann. Statist. 29(4) 1165-1188. The Benjamini-Hochberg procedure controls FDR under positive regression dependence on the true nulls; a conservative modification controls it under any dependence. Abstract-level reading only.

Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, David T. Mellor · Proceedings of the National Academy of Sciences 115(11), 2600-2606 · 2018

Checked 2026-10-12: Search results (PMC6233068 and other indexes): authors, PNAS 115(11) 2600-2606, DOI. Abstract-level reading: hindsight bias makes it hard to separate postdiction from prediction; defining research questions and the analysis plan before observing outcomes (preregistration) helps, and strategies exist when data are pre-existing. Full text not read.

Galit Shmueli · Statistical Science 25(3), from p. 289 · 2010

Explains why explanatory and predictive modeling need different choices of variables, validation and evaluation; core for problem framing.

Checked 2026-10-11: Title, author, Statistical Science vol. 25 issue 3, first page 289, 2010/08 and DOI in citation metadata on the Project Euclid landing page.

John P. A. Ioannidis · PLOS Medicine 2, e124 · 2005

Shows how low prior probability, small effects, flexible analyses and bias raise the false-positive share of findings; motivates multiple-testing and pre-registration discipline.

Checked 2026-10-11: Title, author, PLOS Medicine vol. 2, article e124, Aug 30 2005 and DOI in citation metadata on the PLOS landing page.

Peter van der Putten, Maarten van Someren · Machine Learning 57(1-2), 177-195 · 2004

Analyses why challenge entries on the COIL 2000 data varied, framed as bias and variance; Project 014.

Checked 2026-10-11: Title, authors, Machine Learning 57(1-2) pp. 177-195 in Crossref DOI metadata.

Marco Saerens, Patrice Latinne, Christine Decaestecker · Neural Computation 14(1), pp. 21-41 · 2002

Checked 2026-10-11: Crossref metadata for the DOI: title, three authors, Neural Computation vol. 14 issue 1 pp. 21-41, 2002; abstract: posteriors trained under one class prior are not valid under new priors and can be adjusted with an iterative procedure.

Sudhir Varma, Richard Simon · BMC Bioinformatics 7, article 91 · 2006

Shows the minimum CV error after tuning is optimistically biased and recommends nested cross-validation; Day 15.

Checked 2026-10-11: Title, authors, BMC Bioinformatics vol. 7, Feb 2006 in Crossref DOI metadata (article number from DOI suffix).

Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn · BMC Bioinformatics 8, article 25 · 2007

Impurity-based RF importance is biased toward predictors with many categories / split points; supports caution about default feature_importances_; Day 17.

Checked 2026-10-11: Title, authors, BMC Bioinformatics vol. 8, Jan 2007 in Crossref DOI metadata (article number from DOI suffix).

Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson, Gianluca Bontempi · 2015 IEEE Symposium Series on Computational Intelligence (SSCI), 159-166 · 2015

Undersampling shifts predicted probabilities and how to correct them; dataset citation for Project 010.

Checked 2026-10-11: Title, authors, SSCI 2015 proceedings pp. 159-166 in Crossref DOI metadata; OpenML data 1597 asks users to cite this paper (it names the CIDM symposium, part of SSCI 2015).

Christian Hennig · Computational Statistics & Data Analysis 52(1), 258-271 · 2007

Bootstrap (Jaccard) stability of individual clusters; Project 015.

Checked 2026-10-11: Title, author, CSDA 52(1) pp. 258-271 in Crossref DOI metadata.

Alexandra Chouldechova · Big Data 5(2), 153-163 · 2017

Checked 2026-10-12: arXiv listing 1610.07524 (title, single author, submitted 24 Oct 2016, FATML 2016) and a search record giving Big Data 5(2) 153-163, DOI 10.1089/big.2016.0047. Abstract-level reading: when prevalence of the outcome differs across groups, the constraints of the predictive bias criterion force false negative and false positive rates to differ, and error-rate differences can produce disparate impact. The PDF text layer was not readable here, so theorem numbering was not checked.

Milos Radovanovic, Alexandros Nanopoulos, Mirjana Ivanovic · Journal of Machine Learning Research 11, 2487-2531 · 2010

Checked 2026-10-12: JMLR page: title, three authors, vol. 11 pp. 2487-2531, 2010. Abstract: as dimensionality grows the distribution of k-occurrences becomes skewed and a few hubs appear among the nearest neighbours of many points; theory and experiments on synthetic and real data; an inherent property of high-dimensional distributions.

Sayash Kapoor, Arvind Narayanan · Patterns 4(9), 100804 · 2023

Survey finding leakage in at least 294 studies across 17 fields, with a taxonomy of eight leakage types and 'model info sheets'; good for leakage checklists.

Checked 2026-10-11: Title, authors, Patterns vol. 4 issue 9, article 100804, September 08 2023, open access, on the Cell Press landing page (opened in a browser); matches Crossref DOI metadata.

Olivier Chapelle · Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2014) · 2014

Checked 2026-10-12: Bibliographic data (single author, KDD 2014, pages 1097-1105, DOI 10.1145/2623330.2623634) and the described method (probabilistic delay model deciding whether a not-yet-converted user is a negative example or is excluded from training) confirmed from a web search and the Semantic Scholar entry; the ACM page returned HTTP 403, so the full text was not read.

Gavin C. Cawley, Nicola L. C. Talbot · Journal of Machine Learning Research 11(70), 2079-2107 · 2010

Tuning hyperparameters on the same CV estimate used for reporting yields optimistic bias; model selection itself can overfit; Day 15.

Checked 2026-10-11: Title, authors, JMLR 11 issue 70 pp. 2079-2107, 2010 in citation metadata on JMLR landing page.

Alexandru Niculescu-Mizil, Rich Caruana · Proceedings of the 22nd International Conference on Machine Learning (ICML 2005), 625-632 · 2005

Compares calibration of common learners (e.g., boosted trees, SVMs distort probabilities) and Platt scaling vs isotonic regression; Day 13.

Checked 2026-10-11: Title, authors, ICML '05 proceedings pp. 625-632, 2005 in Crossref DOI metadata.

C. Okan Sakar, S. Olcay Polat, Mete Katircioglu, Yomi Kastro · Neural Computing and Applications 31(10), 6893-6908 · 2019

Dataset paper for Online Shoppers Purchasing Intention; Project 006.

Checked 2026-10-11: Title, authors, Neural Computing and Applications 31(10) pp. 6893-6908 in Crossref DOI metadata (issued online 2018).

Yonatan Geifman, Ran El-Yaniv · arXiv:1705.08500 (cs.LG) · 2017

The coverage versus risk trade-off behind abstention thresholds; supports day 51 and 13.

Checked 2026-10-12: arXiv abstract page: title, two authors, May 2017. Abstract: a trained network can be wrapped in a reject option so that it meets a user-set risk level by rejecting inputs, trading coverage for lower error; one result is 2% top-5 ImageNet error with 99.9% probability at nearly 60% coverage. Image classification; the page does not use the phrase risk-coverage curve.

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith · arXiv:1909.03004 (cs.LG); EMNLP 2019 per the paper's own venue, not shown on the arXiv page · 2019

Supports reporting how many variants or trials were tried alongside the best score.

Checked 2026-10-12: arXiv abstract page (title, 5 authors, v1 6 Sep 2019). Abstract: test scores alone cannot reliably show which model is best; the authors propose reporting expected validation performance of the best model as a function of the budget (hyperparameter trials or training time) and show conclusions that change with more or less compute.

Alexandros Kalousis, Julien Prados, Melanie Hilario · Knowledge and Information Systems 12(1), pp. 95-116 · 2007

Selection stability measures; Day 17.

Checked 2026-10-12: Title, authors, journal, volume, pages and DOI confirmed from search-result records of the open-access repository copy (doc.rero.ch) and a later JMLR paper citing it; abstract summary (stability as sensitivity of feature preferences to training-data changes, comparison of stability measures, stability profiles to choose an algorithm, proteomics experiments). Full text not read.

Ravid Shwartz-Ziv, Amitai Armon · Information Fusion 81, 84-90 · 2022

Tabular deep learning vs XGBoost (days 12, 35).

Checked 2026-10-11: Crossref record for DOI 10.1016/j.inffus.2021.11.011 confirms title, authors, Information Fusion vol 81 pp 84-90, 2022. Abstract (arXiv 2106.03253): XGBoost outperformed the proposed deep tabular models across datasets, needed much less tuning, and an ensemble of both did best.

Stefan Axelsson · ACM Transactions on Information and System Security 3(3), pp. 186-205 · 2000

Checked 2026-10-11: Crossref metadata for the DOI: title, author, journal, vol. 3 issue 3, pp. 186-205, August 2000; abstract: because of the base-rate fallacy the false alarm rate limits detector usefulness, P(intrusion | alarm) needs a very low false alarm rate.

I-Cheng Yeh, Che-hui Lien · Expert Systems with Applications 36(2), 2473-2480 · 2009

Dataset paper; argues estimated default probability matters more than a binary label; Project 011.

Checked 2026-10-11: Title, authors, ESWA 36(2) pp. 2473-2480 in Crossref DOI metadata; cited as the intro paper on the UCI page.

Ruben van den Goorbergh, Maarten van Smeden, Dirk Timmerman, Ben Van Calster · Journal of the American Medical Informatics Association 29(9), 1525-1534 · 2022

Random over/undersampling and SMOTE led to strongly miscalibrated (overestimated) risks without improving discrimination in their study; Day 13.

Checked 2026-10-11: Title, authors, JAMIA 29(9) pp. 1525-1534, 2022 in Crossref DOI metadata.

Takaya Saito, Marc Rehmsmeier · PLOS ONE 10(3), e0118432 · 2015

PR curves reflect performance on the minority class under imbalance where ROC can look optimistic; Day 13 and Day 16 metric choice.

Checked 2026-10-11: Title, authors, PLOS ONE 10(3) e0118432, March 2015 in Crossref DOI metadata.

Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017), 275-284 · 2017

Checked 2026-10-12: Full text opened (author copy at Cornell, text extracted). Abstract and introduction: outcomes are observed only for instances a decision-maker selected, so labelled instances are not a random sample; the contraction technique evaluates a model against human decision-makers by using the heterogeneity of decision-makers, under the assumption that cases are as if randomly assigned across them; demonstrated on criminal justice, health care and insurance data. Venue, pages and DOI 10.1145/3097983.3098066 from a bibliographic record.

Larry R. Weatherford, Sheryl E. Kimes · International Journal of Forecasting 19(3), 401-415 · 2003

Hotel arrivals forecasting with Choice and Marriott data: pickup and regression had lowest error on Choice data; exponential smoothing, pickup and moving average were most robust on Marriott data (abstract); Day 26/28 revenue-management case.

Checked 2026-10-11: Title, author surnames (Weatherford, Kimes), IJF 19(3) pp. 401-415, July 2003 in Crossref bibliographic search result for this DOI.

Ivo Welch, Amit Goyal · Review of Financial Studies 21(4), 1455-1508 · 2008

Limits of financial return forecastability; Day 21/25 insights.

Checked 2026-10-11: Title, authors, RFS 21(4) pp. 1455-1508 in Crossref DOI metadata; RePEc abstract: proposed equity-premium predictors performed poorly both in-sample and out-of-sample over about 30 years, appeared unstable, and would not have helped a real-time investor time the market.

Christoph Bergmeir, Rob J. Hyndman, Bonsoo Koo · Computational Statistics & Data Analysis 120, 70-83 · 2018

Standard K-fold CV is valid for purely autoregressive models when residuals are uncorrelated; nuance to 'never shuffle time series'; Day 25.

Checked 2026-10-11: Title, authors, CSDA vol. 120 pp. 70-83, Apr 2018 in Crossref DOI metadata.

Souhaib Ben Taieb, Gianluca Bontempi, Amir F. Atiya, Antti Sorjamaa · Expert Systems with Applications 39(8), 7067-7083 · 2012

Direct vs recursive vs multi-output strategies; Day 24. The finding is for NN5 daily cash-withdrawal series and the models studied.

Checked 2026-10-11: Title, authors, ESWA 39(8) pp. 7067-7083 in Crossref DOI metadata; arXiv 1108.3259 abstract: 111 NN5 series, multiple-output strategies performed best, deseasonalization uniformly improved accuracy.

Rob J. Hyndman, Anne B. Koehler · International Journal of Forecasting 22(4), 679-688 · 2006

Critiques MAPE/sMAPE/relative measures and proposes MASE, scaling errors by the in-sample (seasonal) naive MAE; Day 16 and Day 25.

Checked 2026-10-11: Title, authors, IJF 22(4) pp. 679-688, Oct 2006 in Crossref DOI metadata.

Christoph Bergmeir, Rob J. Hyndman, Jose M. Benitez · International Journal of Forecasting 32(2), 303-312 · 2016

Bagged ETS via STL-remainder bootstrap; cite the FPP3 section for the procedure and this paper for the reported M3 result.

Checked 2026-10-12: Title, venue, volume, pages and DOI via IDEAS/RePEc listing returned by search (v32y2016i2p303-312); abstract summary: Box-Cox plus STL, moving-block bootstrap of the remainder, ensemble of exponential smoothing forecasts, consistent gains over the original ETS on the M3 data. Publisher full text not read. FPP3 section 'Bootstrapping and bagging' (otexts.com/fpp3/bootstrap.html) was opened and states the procedure and that bagging gave better forecasts than ETS alone on average, at much higher compute cost.

L. R. Weatherford, S. Pölt · Journal of Revenue and Pricing Management 1(3), 234-254 · 2002

Compares methods for unconstraining capacity-limited airline booking data; supports the claim that observed bookings understate demand when flights close.

Checked 2026-10-11: Title, authors, JRPM 1(3) pp. 234-254, Oct 2002 in Crossref DOI metadata.

Johannes Bracher, Evan L. Ray, Tilmann Gneiting, Nicholas G. Reich · PLOS Computational Biology 17(2), e1008618 · 2021

Checked 2026-10-12: Crossref metadata and the PLOS article page: the weighted interval score is a proper score for forecasts given as a median and central intervals, combines sharpness (width) and penalties for observations outside intervals, approximates CRPS with many equally spaced levels and equals absolute error for a median-only forecast; PIT histograms recommended for calibration checks.

Stephan Kolassa · International Journal of Forecasting 32(3), 788-803 · 2016

Checked 2026-10-12: Crossref metadata and the RePEc abstract: low counts at fine granularity call for full discrete predictive distributions; MAD, MASE and wMAPE are argued to be inherently unsuitable for count data; evaluation uses the randomized PIT and proper scoring rules on two daily retail datasets.

Vitor Cerqueira, Luis Torgo, Igor Mozetič · Machine Learning 109(11), 1997-2028 · 2020

174 real series + synthetic: blocked CV works for stationary series, but for non-stationary series out-of-sample methods (repeated holdout over multiple test periods) gave the most accurate estimates (abstract); Day 25.

Checked 2026-10-11: Title, authors, Machine Learning 109(11) pp. 1997-2028, 2020 in Crossref DOI metadata.

Alysha M. De Livera, Rob J. Hyndman, Ralph D. Snyder · Journal of the American Statistical Association 106(496), 1513-1527 · 2011

BATS/TBATS: exponential smoothing with multiple (including non-integer) seasonal periods via trigonometric terms; multiple-seasonality project.

Checked 2026-10-11: Title, authors, JASA 106(496) pp. 1513-1527, Dec 2011 in Crossref DOI metadata.

George Athanasopoulos, Rob J. Hyndman, Nikolaos Kourentzes, Fotios Petropoulos · European Journal of Operational Research 262(1), 60-74 · 2017

Checked 2026-10-12: Bibliographic data (EJOR 262(1) 60-74, DOI 10.1016/j.ejor.2017.02.046) from a library-portal record. Read the 2015 working-paper PDF (MPRA 66362, text extracted): abstract (temporal hierarchy from non-overlapping temporal aggregation; predictions at all levels combined into temporally reconciled forecasts; mitigates modelling uncertainty; accuracy gain over conventional forecasting, particularly under increased modelling uncertainty) and Section 3 (monthly data aggregated at k in {12, 6, 4, 3, 2, 1}; evaluation on 1,428 monthly and 756 quarterly M3 series). The journal text was not opened, so claims are limited to the abstract.

Fotios Petropoulos, Daniele Apiletti, Vassilios Assimakopoulos, et al. (80 authors) · International Journal of Forecasting 38(3), 705-871 · 2022

Encyclopedic review of forecasting methods and applications; reference map for Days 21-28.

Checked 2026-10-11: Title, full author list (first author Petropoulos), IJF 38(3) pp. 705-871, July 2022 in Crossref DOI metadata.

Ruud H. Teunter, Aris A. Syntetos, M. Zied Babai · European Journal of Operational Research 214(3), 606-615 · 2011

TSB method: updates the demand probability every period (not only after a demand), so forecasts decay for items that stop selling; intermittent-demand project.

Checked 2026-10-11: Title, authors, EJOR 214(3) pp. 606-615, Nov 2011 in Crossref DOI metadata.

Spyros Makridakis, Evangelos Spiliotis, Vassilios Assimakopoulos · International Journal of Forecasting 38(4), 1346-1364 · 2022

Official M5 Accuracy results: 42,840 hierarchical Walmart unit-sales series; best-performing methods and findings; Day 24 ML forecasting (check the paper before attributing wins to specific packages).

Checked 2026-10-11: Title, authors, IJF 38(4) pp. 1346-1364, Oct 2022 in Crossref DOI metadata.

Andrey Davydenko, Robert Fildes · International Journal of Forecasting 29(3), 510-522 · 2013

Checked 2026-10-12: Crossref metadata and the RePEc abstract: recommends aggregating performance ratios across series by the weighted geometric mean, described as treating over- and under-forecasting evenly, symmetric in distribution and robust. The exact ratio definition (relative MAE) was not confirmed from the abstract.

A. A. Syntetos, J. E. Boylan, J. D. Croston · Journal of the Operational Research Society 56(5), 495-503 · 2005

Source for routing intermittent series between Croston-type methods by interval and size variability; do not state numeric cut-offs from this entry.

Checked 2026-10-12: Title, three authors, JORS 56(5) pp. 495-503, May 2005 in Crossref DOI metadata. Abstract content (classification rules stated in terms of the average inter-demand interval and the squared coefficient of variation of demand sizes, derived by comparing forecasting methods on theoretical error measures and tested on 3,000 real intermittent automotive series) taken from the Salford repository record as returned by a search summary; the publisher and repository pages returned HTTP 403 to direct fetches. The numeric cut-off values often quoted for the scheme were NOT confirmed from a primary source.

Christoph Bergmeir, José M. Benítez · Information Sciences 191, 192-213 · 2012

Empirical comparison of CV, blocked CV and last-block holdout for time series predictor evaluation; Day 25.

Checked 2026-10-11: Title, authors, Information Sciences vol. 191 pp. 192-213, May 2012 in Crossref DOI metadata.

Shanika L. Wickramasuriya, Berwin A. Turlach, Rob J. Hyndman · Statistics and Computing 30(5), 1167-1182 · 2020

Checked 2026-10-12: Crossref metadata and the author's publication page (robjhyndman.com): MinT and variants do not guarantee non-negative coherent forecasts; non-negativity constraints are added and a quadratic programme solved with three algorithms; tourism study showed slight gains at the most disaggregated level and slight losses at aggregate levels.

Leonard J. Tashman · International Journal of Forecasting 16(4), 437-450 · 2000

Reviews holdout design for forecast evaluation: rolling origins, fixed vs rolling windows, multiple horizons; Day 25.

Checked 2026-10-11: Title, author, IJF 16(4) pp. 437-450, Oct 2000 in Crossref DOI metadata.

Pablo Montero-Manso, Rob J. Hyndman · International Journal of Forecasting 37(4), 1632-1653 · 2021

Global vs local forecasting models; Day 24.

Checked 2026-10-11: Title, authors, IJF 37(4) pp. 1632-1653 in Crossref DOI metadata; arXiv 2008.00444 abstract: global methods are not more restrictive than local ones, keep constant complexity as the number of series grows, and performed well even on heterogeneous datasets.

Tilmann Gneiting · International Journal of Forecasting 27(2), 197-207 · 2011

Choosing a forecast quantile from asymmetric costs (newsvendor-style decisions); Day 26 decision economics.

Checked 2026-10-11: Title, author, IJF 27(2) pp. 197-207 (2011) in Crossref DOI metadata; abstract on RePEc (ideas.repec.org) states quantiles are optimal point forecasts under generalized piecewise linear losses, which nest asymmetric piecewise linear loss, with the quantile level set by the possibly distinct costs of under- and overprediction.

Aris A. Syntetos, John E. Boylan · International Journal of Forecasting 21(2), 303-314 · 2005

Shows Croston's estimator is biased and evaluates the bias-corrected Syntetos-Boylan Approximation (SBA); Day 26 intermittent demand.

Checked 2026-10-11: Title, authors, IJF 21(2) pp. 303-314, April 2005 in Crossref DOI metadata.

Spyros Makridakis, Evangelos Spiliotis, Vassilios Assimakopoulos · International Journal of Forecasting 36(1), 54-74 · 2020

Official M4 results paper: 100,000 series, 61 methods, point forecasts and prediction intervals; top methods and findings by category; benchmark context for Days 23-24 (read the findings section before quoting specific rankings).

Checked 2026-10-11: Title, authors, IJF 36(1) pp. 54-74, Jan 2020 in Crossref DOI metadata.

Miguel A. Hernan, Sonia Hernandez-Diaz · Clinical Trials 9(1), 48-55 · 2012

Checked 2026-10-12: SAGE record and PMC3731071 via search results: title, authors, Clin Trials 9(1) 48-55 (online Sep 2011). Content: the intention-to-treat comparison is not guaranteed to estimate effectiveness and can understate an effect when adherence differs; recommends reporting as-treated or per-protocol analyses with weighting adjustments that rely on untestable assumptions. Abstract-level reading only.

Felix Elwert, Christopher Winship · Annual Review of Sociology · 2014

Checked 2026-10-11: Semantic Scholar record for the DOI: title, authors, venue Annual Review of Sociology, 2014, abstract. Abstract: endogenous selection bias arises from conditioning (controlling, stratifying or selecting) on a collider caused by two variables, one associated with the treatment and one with the outcome; uses DAGs to show it. Volume and pages not checked.

Douglas Staiger, James H. Stock · Econometrica 65(3), 557-586 · 1997

Checked 2026-10-12: Econometric Society listing and search summaries: title, authors, Econometrica 65(3) May 1997. Abstract: asymptotic theory for IV regression when instruments are only weakly correlated with the endogenous regressor; two-stage least squares with many weak instruments is pulled toward OLS in the Angrist-Krueger illustration. Abstract-level reading only.

James Lopez Bernal, Steven Cummins, Antonio Gasparrini · International Journal of Epidemiology 46(1), 348-355 · 2017

Checked 2026-10-11: Oxford Academic article page: title, three authors, IJE 46(1) 348-355 (Feb 2017, online June 2016). Content: seasonality can bias estimates when pre and post periods contain uneven seasonal months (adjust with month terms, Fourier terms or splines); ITS is protected against slowly changing confounders but events coinciding with the intervention can bias it, addressed by controlled ITS or other design extensions; check residual autocorrelation; no fixed minimum number of time points.

Justin McCrary · Journal of Econometrics 142(2), 698-714 · 2008

Checked 2026-10-12: NBER working-paper version (t0334) abstract via search results: identification relies on continuity of counterfactual outcomes in the running variable, which may fail if agents can manipulate it; the paper tests continuity of the running-variable density at the cutoff. Journal citation (J. Econometrics 142(2) 698-714) from the same results; journal page not opened.

Peter C. Austin, Elizabeth A. Stuart · Statistics in Medicine 34(28), 3661-3679 · 2015

Checked 2026-10-12: PubMed Central record (via search results): title, authors, Stat Med 34(28) 3661-3679, 2015, DOI 10.1002/sim.6607. Abstract: IPTW creates a pseudo-sample in which treatment assignment no longer depends on measured covariates; the paper proposes qualitative and quantitative balance diagnostics for the weighted sample. Abstract-level reading only.

Tyler J. VanderWeele, Peng Ding · Annals of Internal Medicine 167(4), 268-274 · 2017

Checked 2026-10-12: Harvard DASH manuscript record and search summaries: title, authors, Ann Intern Med 167(4) 268-274. Abstract: the E-value is the minimum strength of association (risk-ratio scale) an unmeasured confounder would need with both treatment and outcome to fully explain away an association, given measured covariates; it should be reported for the estimate and the confidence limit closest to the null.

Alberto Abadie · Journal of Economic Literature 59(2), 391-425 · 2021

Checked 2026-10-11: AEA landing page (title, author, JEL 59(2) 391-425, June 2021) and the full text (NBER-hosted PDF): when pre-intervention fit is poor the method is not recommended because of potential bias; a long pre-period cannot remove bias if the fit is bad; over-fitting risk grows with donor pool size when the pre-period is short; donors must be chosen judiciously; backdating (in-time placebo) checks credibility.

Jonathan Roth, Pedro H. C. Sant'Anna, Alyssa Bilinski, John Poe · Journal of Econometrics (2023); arXiv:2201.01194 · 2023

Checked 2026-10-11: arXiv page and v3 PDF (9 Jan 2023): no significant pre-trend does not imply parallel trends, pre-tests often have low power, conditioning on passing a pre-test causes pre-test bias; with staggered timing and heterogeneous effects TWFE can make 'forbidden' comparisons with already-treated units and even get the sign wrong. Journal of Econometrics venue from the Semantic Scholar record for DOI 10.1016/j.jeconom.2023.03.008 (journal page returned 403).

Andrew Gelman, Guido Imbens · Journal of Business & Economic Statistics 37(3), 447-456 (NBER Working Paper 20405, 2014) · 2019

Checked 2026-10-12: NBER working-paper page and search summaries: title, authors, JBES 37(3) 447-456 per IDEAS/RePEc (the NBER page cites DOI 10.1080/07350015.2017.1366909). Argument: estimators controlling for high-order (third, fourth or higher) polynomials of the forcing variable can be misleading, and local linear or quadratic estimators are recommended. Abstract-level reading only.

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal · Nature 631(8022), 755-759 · 2024

Indiscriminately training on model-generated content causes irreversible defects in which tails of the original distribution disappear (shown for LLMs, VAEs and Gaussian mixtures). Supports insight 0545.

Checked 2026-10-11: Crossref record for DOI 10.1038/s41586-024-07566-y (title, authors, journal, volume/issue/pages, online 2024-07-24, abstract). A correction was published 2025-03-21 (DOI 10.1038/s41586-025-08905-3).

Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu · Proceedings of the AAAI Conference on Artificial Intelligence 37(9), 11121-11128 · 2023

Simple linear models (DLinear/NLinear) outperformed several Transformer LTSF models on standard benchmarks: a baseline lesson; Day 27.

Checked 2026-10-11: Title, authors, AAAI 37(9) pp. 11121-11128, 2023 in Crossref DOI metadata; arXiv 2205.13504 lists code at github.com/cure-lab/LTSF-Linear.

Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang · Journal of Artificial Intelligence Research (JAIR), 2021 (per arXiv journal reference) · 2021

Checked 2026-10-12: arXiv abstract page: title, three authors, submitted 31 Oct 2019, journal reference JAIR 2021. Abstract: confident learning finds label errors by pruning suspect examples, estimating noise with thresholded counting and ranking examples, under a class-conditional noise assumption; reported to improve accuracy after cleaning.

Gael Varoquaux · arXiv:1706.07581 (the paper appeared in NeuroImage; journal details not shown on the arXiv page) · 2017

Checked 2026-10-12: arXiv abstract page: title, single author, submitted 23 Jun 2017. Abstract: cross-validation error bars are often underestimated; for typical neuroimaging sample sizes (about 100 samples) the uncertainty is on the order of +-10% and is not captured by the spread across folds. Journal venue not confirmed from the page.

Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217-2226 · 2019

Checked 2026-10-11: Crossref metadata for the DOI: title, four authors, journal, volume 12 issue 7 pages 2217-2226, July 2019. arXiv 1709.00029 abstract: 13 bands, 10 classes, 27,000 images, CNN benchmarks, overall accuracy 98.57% reported by the authors.

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar · Journal of Machine Learning Research 18 (2018) 1-52 · 2018

Checked 2026-10-12: arXiv abstract page: title, five authors, submitted 21 Mar 2016, journal reference JMLR 18 (2018) 1-52. Abstract: speeds up random search by adaptively allocating resources and early-stopping poor configurations; framed as a pure-exploration bandit problem; reported to beat Bayesian optimization methods on deep learning and kernel tasks.

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox, Hugo Larochelle · JMLR (arXiv comment: 'To appear at JMLR') · 2021

Checked 2026-10-12: arXiv abstract page: title, eight authors, submitted 27 Mar 2020, comment 'To appear at JMLR'. Abstract: reports the NeurIPS 2019 reproducibility programme of code submission, a community reproducibility challenge and a reproducibility checklist for papers. Volume/pages not confirmed.

Aditya Kusupati et al. · arXiv:2205.13147 (cs.LG) · 2022

Why some embedding models can be shortened by dropping trailing dimensions; supports day 38 and 48.

Checked 2026-10-12: arXiv abstract page: title, eleven authors, submitted May 2022 (v4 Feb 2024). Abstract: representations encode information at multiple granularities so one embedding can be truncated to a task's budget; reports up to 14x smaller embeddings at the same ImageNet-1K accuracy and up to 14x faster large-scale retrieval. Vision-centred results.

Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl · Journal of Machine Learning Research 20 (2019) 1-49 · 2019

Checked 2026-10-12: arXiv abstract page: title, six authors, submitted 8 Nov 2018, journal reference JMLR 20 (2019) 1-49. Abstract: empirically measures how larger batches change the number of steps needed to reach a target out-of-sample error across workloads; the effect varies greatly between workloads; conflicting prior findings often trace to tuning and compute budgets; no evidence that larger batches hurt generalization was found.

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger · Proceedings of the 34th International Conference on Machine Learning (ICML 2017), PMLR 70, 1321-1330 · 2017

Modern deep nets are often overconfident; temperature scaling is a simple effective post-hoc fix; ECE and reliability diagrams; Day 13, Day 34.

Checked 2026-10-11: Title, authors, pp. 1321-1330, conference ICML, PMLR v70 in citation metadata on PMLR landing page.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. · Proceedings of the National Academy of Sciences 114(13), 3521-3526 · 2017

Catastrophic forgetting (days 40, 44).

Checked 2026-10-11: Crossref record for DOI 10.1073/pnas.1611835114 confirms title, PNAS vol 114 issue 13 pp 3521-3526, March 2017; arXiv 1612.00796 confirms authors. Abstract: networks generally forget earlier tasks when trained sequentially; EWC slows learning on weights important to old tasks.

Yantao Shen, Yuanjun Xiong, Wei Xia, Stefano Soatto · CVPR 2020 (arXiv:2003.11942) · 2020

Why a model update means re-embedding the whole gallery unless the new model was trained to be compatible; supports day 38 and 48.

Checked 2026-10-12: arXiv abstract page: title, four authors, 2020, accepted to CVPR 2020. Abstract: updating an embedding model normally forces recomputing ('backfilling') embeddings of all previously seen items because new and old features are not directly comparable; backward-compatible training is proposed so old and new features can be compared without backfill. Image-retrieval setting, not text; abstract-level reading.

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams · NAACL 2021 (arXiv:2104.14337) · 2021

Checked 2026-10-12: arXiv abstract page (title, 19 authors, v1 7 Apr 2021, NAACL 2021 noted). Abstract: models reach strong benchmark scores quickly yet still fail simple challenge examples; Dynabench collects new examples with annotators writing items that a target model gets wrong but a person answers correctly.

Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen · Proceedings of EMNLP 2023, 6465-6488 · 2023

Citation-quality evaluation for retrieval-augmented answers; supports days 48, 51 and 57.

Checked 2026-10-11: ACL Anthology 2023.emnlp-main.398 (via search listing) and arXiv 2305.14627 comment 'Accepted by EMNLP 2023'. Abstract: ALCE benchmark scores fluency, correctness and citation quality; on ELI5 even the best models lack complete citation support about half the time.

Nelson F. Liu, Tianyi Zhang, Percy Liang · Findings of the Association for Computational Linguistics - EMNLP 2023, 7001-7025 · 2023

Citation recall vs citation precision; supports days 51 and 57.

Checked 2026-10-11: arXiv 2304.09848 abstract and comment (Findings of EMNLP 2023); Anthology id 2023.findings-emnlp.467 with pages 7001-7025 from the Anthology BibTeX listing. Human audit of four generative search engines: about 51.5% of generated sentences fully supported by citations, about 74.5% of citations support their sentence; defines citation recall and citation precision.

Angela Fan, Mike Lewis, Yann Dauphin · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898 · 2018

Early, widely cited use of top-k sampling for text generation (Day 42).

Checked 2026-10-11: ACL Anthology page confirms title, three authors, ACL 2018 long papers, pages 889-898, DOI 10.18653/v1/P18-1082. The arXiv PDF (1805.04833), Section 5.4 'Generation', describes sampling from the k = 10 most likely candidates at each step (top-k random sampling) and tuning a softmax temperature at generation time; the abstract itself does not mention top-k.

Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna · COLING 2020 (arXiv:2010.14571) · 2020

Web-corpus building for lower-resource languages, not chat routing. Supports only the point that held-out LangID scores can overstate accuracy on in-the-wild text; short-text and code-mixing effects are not in the abstract.

Checked 2026-10-12: arXiv abstract page: title, four authors, COLING 2020, submitted 27 Oct 2020. Abstract: language identification is often treated as solved, with reported F1 above 90% over up to 1,366 languages; the authors trained models for up to 1,629 languages with comparable held-out quality, yet human-judged accuracy on web-crawl corpora built with them was only about 5% for many lower-resource languages; error sources listed include language similarity, domain mismatch, class imbalance and under-expressive models; wordlist filters and a semi-supervised transformer LangID raised median dataset precision from 5.5% to 71.2%. The abstract does not discuss short texts.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, Minlie Huang · ICLR 2024 (arXiv:2309.03882) · 2024

Abstract-level reading only. Not the same paper as paper-zheng-2023-llm-as-judge (pairwise judge position bias).

Checked 2026-10-12: arXiv abstract page: title, five authors, ICLR 2024 (spotlight), submitted 7 Sep 2023, v4 22 Feb 2024. Abstract: LLMs are sensitive to the position of options in multiple-choice questions (selection bias); they tend to prefer particular option IDs such as 'Option A'; the authors attribute this mainly to token bias in option-ID tokens; PriDe estimates the option-ID prior by permuting option contents on a small number of samples and debiases the rest; analyses cover 20 LLMs and three benchmarks. The abstract gives no numeric results.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, Denny Zhou · Proceedings of the 40th International Conference on Machine Learning (PMLR 202), 31210-31227 · 2023

Irrelevant context degrades LLM reasoning; supports days 48 and 51 (filter weak retrieved chunks).

Checked 2026-10-11: PMLR v202 page: title, eight authors, ICML 2023, pages 31210-31227. arXiv 2302.00093 abstract: accuracy on GSM-IC drops sharply when irrelevant information is added; self-consistency and an instruction to ignore irrelevant information mitigate.

Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio · Journal of Artificial Intelligence Research 72 (2021), DOI 10.1613/jair.1.12752 · 2021

Checked 2026-10-12: JAIR article page: title, six authors, vol. 72, published 27 Dec 2021, DOI. Abstract: relative performance of training methods depends heavily on how models are evaluated and there is a strong dataset effect; with large datasets and many judgments per item from high-quality annotators, training on soft labels (the distribution of judgments) beats training on aggregated or gold labels under hard and soft evaluation; otherwise combining gold and soft labels generally did best under hard evaluation. Page lists no page range.

Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer · NeurIPS 2022 (arXiv:2208.07339) · 2022

Checked 2026-10-12: arXiv abstract page: title, four authors, v1 15 Aug 2022, published at NeurIPS 2022. Abstract: transformers have systematic emergent outlier features that dominate attention and predictive performance; vector-wise quantisation handles most features in 8-bit while the outlier dimensions go through a 16-bit matrix multiplication, so more than 99.9% of values are multiplied in 8-bit; a 175B model can be converted to Int8 without performance degradation and about half the inference memory.

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu · EMNLP 2023 (per the arXiv page); arXiv 2310.05736 · 2023

Supports insight 1521. Authors' datasets only.

Checked 2026-10-12: arXiv abstract page opened 2026-10-12: title, five authors, first submitted 2023-10-09, revised 2023-12-06, accepted at EMNLP 2023. Abstract: coarse-to-fine compression with a budget controller, token-level iterative compression and distribution alignment; evaluated on GSM8K, BBH, ShareGPT and Arxiv-March23; up to 20x compression with little performance loss. Full text not read, so per-task losses are unknown.

Orion Weller, Dawn Lawrie, Benjamin Van Durme · EACL 2024 (arXiv:2305.07614) · 2024

Abstract-level reading; the benchmark documents are general-domain, not policy text.

Checked 2026-10-12: arXiv abstract page: title, three authors, EACL 2024, submitted 12 May 2023, v2 26 Feb 2024. Abstract: a benchmark that asks retrievers to rank two documents that differ only by negation; cross-encoders performed best, then late-interaction models, with bi-encoders and sparse neural models last; most models, including state-of-the-art ones, performed the same as or worse than random ranking; fine-tuning on contrastive negation data and larger size helped but a large gap to humans remained.

Stephen Casper et al. (32 authors) · Transactions on Machine Learning Research (TMLR), 2023 · 2023

Survey of open problems and fundamental limitations of RLHF across human feedback, the reward model and the policy. Supports insight 0566.

Checked 2026-10-11: arXiv abs page (v2 2023-09-11) and abstract; venue confirmed on the official TMLR accepted-papers list (Survey certification). Section 3 of the PDF groups challenges by human feedback, reward model, policy and joint training.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, Nan Duan · EMNLP 2023 (arXiv:2305.14283) · 2023

Evidence that query rewriting can help retrieval-augmented LLMs; supports day 48.

Checked 2026-10-12: arXiv abstract page: title, five authors, May 2023 (revised Oct 2023). Abstract: the input text and the knowledge needed for retrieval often differ, so the query is rewritten (by an LLM, then a small trained rewriter) before retrieval; consistent gains on open-domain and multiple-choice QA. Single-turn QA benchmarks, not conversations.

Gordon V. Cormack, Charles L. A. Clarke, Stefan Buettcher · Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 758-759 · 2009

Reciprocal rank fusion combines rankings by rank position rather than raw scores; supports day 48 hybrid retrieval.

Checked 2026-10-11: Crossref record for DOI 10.1145/1571941.1572114: title, three authors, SIGIR 2009 proceedings, published 2009-07-19, pages 758-759. ACM landing page returned 403, so the abstract was not read there.

Ruiqi Zhong, Tao Yu, Dan Klein · EMNLP 2020 (arXiv:2010.02840) · 2020

Spider-specific 2020 measurements; the false-positive rate depends on the benchmark and models and is not a general constant. Abstract-level reading plus an HTML rendering of the body; the insight relies only on the mechanism and the abstract's numbers.

Checked 2026-10-12: arXiv abstract page opened 2026-10-12: title, three authors, EMNLP 2020, v1 submitted 6 Oct 2020. Abstract: proposes test suite accuracy to approximate semantic accuracy for Text-to-SQL; distils a small test suite of databases with high code coverage for the gold query from a large number of randomly generated databases; computes denotation accuracy of the predicted queries on that suite, a tight upper bound for semantic accuracy; evaluated 21 models submitted to the Spider leaderboard and manually verified correctness on 100 examples; the Spider metric then gave a 2.5% false negative rate on average and 8.1% in the worst case; code and suites for eleven datasets released. Body text read through the ar5iv HTML rendering (summarised by the fetch tool, not re-read in the PDF): single-database denotation accuracy can give false positives, where a semantically different prediction happens to return the gold result on that particular database; the paper reports a mean false-positive rate of about 6.5% and suites of about 42 databases per gold query distilled from 1,000 random databases. Those body figures are NOT used in the insight.

Ron Artstein, Massimo Poesio · Computational Linguistics 34(4), 555-596 · 2008

Checked 2026-10-12: ACL Anthology page (title, authors, volume 34 number 4, pages 555-596, DOI 10.1162/coli.07-034-R2) and PDF text. Full text: observed agreement must be adjusted for chance agreement to be comparable across studies; chance-corrected coefficients (kappa, pi, alpha) differ in how chance agreement is estimated; Section 3 discusses bias and prevalence, i.e. how highly skewed category distributions make high coefficient values hard to reach.

Ziwei Ji et al. · ACM Computing Surveys 55(12), 1-38 · 2023

Taxonomy of intrinsic vs extrinsic hallucination and mitigation families; supports days 45 and 48 (why grounding and evaluation are needed).

Checked 2026-10-11: Crossref record for DOI 10.1145/3571730 confirms title, first author Ji, ACM Computing Surveys vol 55 issue 12 pp 1-38 (issue dated 2023). arXiv 2202.03629 links the same DOI. Author count differs between Crossref and arXiv, so authors given as "et al.".

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, Fabrizio Silvestri · arXiv:2401.14887 (cs.CL) · 2024

Hard negatives matter more than random ones when testing RAG robustness; supports day 51.

Checked 2026-10-12: arXiv abstract page: title, eight authors, January 2024. Abstract: top-ranked documents that are related but contain no answer reduce LLM effectiveness, while adding random documents raised accuracy by up to 35% in their setup. Counter-intuitive random-document result is specific to their models and datasets.

Ellen M. Voorhees, Dawn M. Tice · Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000) · 2000

Origin of mean reciprocal rank (MRR) as a QA/retrieval score; supports day 51.

Checked 2026-10-11: ACL Anthology L00-1018: title, two authors, LREC 2000. PDF text: each question scored as the reciprocal of the rank of the first correct response among five, or 0 if none; the run score is the mean (mean reciprocal rank).

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, Dirk Hovy · NAACL 2024 (accepted; arXiv 2308.01263) · 2024

Supports insight 1511.

Checked 2026-10-12: arXiv abstract page opened 2026-10-12: title, six authors, submitted 2023-08, revised 2024-04, accepted at NAACL 2024. Abstract: 250 safe prompts across ten prompt types that well-calibrated models should not refuse, plus 200 unsafe contrast prompts; exaggerated safety is refusing clearly safe prompts because of similar language to unsafe prompts or sensitive topics; used to identify systematic failure modes in state-of-the-art models. Full text not read.

Neoklis Polyzotis, Martin Zinkevich, Sudip Roy, Eric Breck, Steven Whang · Proceedings of Machine Learning and Systems 1 (MLSys 2019) · 2019

Checked 2026-10-11: MLSys proceedings abstract page: title, authors, 2019; data validation system deployed in TFX; abstract names training/serving skew and unexpected data patterns as key challenges.

Stephan Rabanser, Stephan Günnemann, Zachary Lipton · Advances in Neural Information Processing Systems 32 (NeurIPS 2019) · 2019

Shift detection is a two-sample testing problem; detecting a shift is separate from judging whether it is harmful. Drift-monitoring project.

Checked 2026-10-11: NeurIPS proceedings page: title, authors, venue; abstract states two-sample testing on representations from pre-trained classifiers performed best among the tested detectors, and domain-discriminating approaches help characterise shifts and judge harm.

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun · 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI '22) · 2022

Origin of continuous (iteration-level) batching; the 36.9x figure is the authors' benchmark, not a general expectation.

Checked 2026-10-12: USENIX presentation page: title, five authors, OSDI '22. Abstract: iteration-level scheduling (the batch can change between model iterations, so finished requests return early and new requests need not wait for the whole batch) and selective batching; reports 36.9x throughput at the same latency over NVIDIA FasterTransformer on a GPT-3 175B model.

Jeffrey Dean, Luiz Andre Barroso · Communications of the ACM 56(2) · 2013

Checked 2026-10-12: Google Research publication page: title, authors, Communications of the ACM vol. 56, pp. 74-80, 2013; abstract states that latency tails get harder to control as systems and utilization grow, that brief slowdowns harmless at small scale can dominate at large scale, and proposes latency tail-tolerant techniques. The full text was not read, so the fan-out mechanism itself is not claimed from this source.

William S. Cleveland, Robert McGill · Journal of the American Statistical Association 79(387), 531-554 · 1984

Perceptual ordering of encodings, Stevens' power law and Weber's law (Day 8).

Checked 2026-10-18: Crossref record for the DOI (title, both authors, JASA vol. 79 no. 387, pp. 531-554, September 1984). Scanned article read: section 3 orders the elementary tasks from most to least accurate (position along a common scale; position along nonaligned scales; length, direction, angle; area; volume, curvature; shading, color saturation); p. 537 states Stevens' power law p = k a^alpha with alpha close to 1 for length, smaller for area and smaller still for volume; p. 538 states Weber's law d_p(x) = k_p x.

Justin Matejka, George Fitzmaurice · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 1290-1294 · 2017

Checked 2026-10-12: Autodesk Research publication page (2026-10-12): title, authors, CHI 2017, honorable mention; abstract describes generating datasets with the same statistical properties that look very different when graphed, by small random changes guided by simulated annealing. Crossref confirms DOI, title, authors, pp. 1290-1294.

Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, Matthew Kay · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · 2018

Checked 2026-10-12: Authors' project page: incentivised experiment where participants chose when to catch a bus using textual uncertainty, uncertainty visualisations or a no-uncertainty control; quantile dotplots gave the best and most consistent decisions, CDF plots almost as good, textual uncertainty depended on the probability range it communicated. DOI 10.1145/3173574.3173718 from a CHI listing. Single-domain (bus arrival) experiment.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III · Communications of the ACM 64(12), 86-92 · 2021

Documenting dataset motivation, composition, collection and recommended uses.

Checked 2026-10-11: Crossref DOI metadata: title, CACM 64(12) pp. 86-92, 2021 (authors as listed in Crossref).

Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, Sam Whittle · Proceedings of the VLDB Endowment 8(12) · 2015

Checked 2026-10-11: Read the VLDB PDF: title, authors, PVLDB Vol. 8 No. 12 (2015). Section on time domains: event time vs processing time with dynamically changing skew; watermarks are heuristic lower bounds on processed event times and can be too fast, so late data can arrive behind them; argues one never knows if all data has been seen and results may need refinement or retraction.

Hadley Wickham · Journal of Statistical Software 59(10) · 2014

Checked 2026-10-11: JSS article page: title, author, volume 59 issue 10, published 2014-09-12, DOI 10.18637/jss.v059.i10; tidy definition (variable per column, observation per row, observational unit per table).

Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr · Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track · 2024

Measures both utility and attack success for tool-using agents; supports day 53 red teaming.

Checked 2026-10-11: NeurIPS 2024 proceedings page: title, six authors, Datasets and Benchmarks Track. arXiv 2406.13352 abstract: 97 realistic tasks and 629 security test cases; tool-returned data can hijack agents; models fail many tasks even without attacks; existing attacks break some security properties but not all.

Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, Tatsunori Hashimoto · ICML 2025 (arXiv:2502.07776) · 2025

Audit as of the paper's 2025 measurements; provider behaviour changes, so re-check the provider's current documentation.

Checked 2026-10-12: arXiv abstract page: title, five authors, ICML 2025, submitted 11 Feb 2025, v2 13 Jul 2025. Abstract: prompt caching makes cached prompts faster, so timing differences can enable side-channel attacks and, if the cache is shared across users, let an attacker infer others' prompts; the authors ran statistical audits on real LLM API providers and found global cache sharing across users in seven providers, including OpenAI; timing can also reveal model architecture details (evidence that one embedding model is decoder-only).

Raja Parasuraman, Dietrich Manzey · Human Factors 52(3), 381-410 · 2010

Review of empirical studies on over-reliance on automation; relevant to human approval steps for AI output.

Checked 2026-10-12: Publisher page returned HTTP 403; bibliographic data (title, authors, Human Factors 52(3), pp. 381-410, 2010, DOI) and abstract findings confirmed through search results indexing the SAGE page and the TU Berlin repository: complacency and automation bias occur in naive and expert participants and are not prevented by simple practice or instructions. Full text not read.

Latanya Sweeney · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10(5), 557-570 · 2002

Checked 2026-10-12: Search results and the Data Privacy Lab project pages: journal, volume, issue, pages, year. Abstract-level reading: a release provides k-anonymity protection if the information for each person cannot be distinguished from that of at least k-1 other individuals in the release; de-identified fields (such as diagnosis, gender, date of birth) can be linked to identified sources; the paper discusses attacks that still work against k-anonymous releases unless policies are followed. Full text not read by the author of this record.

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru · Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* 2019), 220-229 · 2019

Checked 2026-10-11: Crossref DOI metadata (title, venue, pages 220-229); arXiv 1810.03993 abstract: short documents reporting performance across relevant conditions and groups, intended use contexts and evaluation procedure.

John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander M. Rush · EMNLP 2023 (arXiv:2310.06816) · 2023

Embedding vectors can leak source text; supports day 53.

Checked 2026-10-12: arXiv abstract page: title, four authors, October 2023, accepted at EMNLP 2023. Abstract: a multi-step correct-and-re-embed method recovers 92% of 32-token text inputs exactly from their embeddings and can recover full names in a clinical-notes dataset. Results are for the models and short inputs they studied.

Raja Parasuraman, Dietrich H. Manzey · Human Factors 52(3), 381-410 · 2010

Checked 2026-10-12: Search results (SAGE, TU Berlin repository, TRID): authors, journal, volume/pages, DOI. Abstract-level reading: review of empirical studies; automation bias yields omission and commission errors when the decision aid is imperfect, occurs in naive and expert users, is not prevented by simple practice, and resists training and instructions. Full text not read.

James E. Smith, Robert L. Winkler · Management Science 52(3), 311-322 · 2006

Checked 2026-10-12: IDEAS/RePEc and Duke faculty listings via search results: title, authors, Management Science 52(3) 311-322. Content: choosing the option with the highest value estimate leaves the decision maker with a result that is expected to fall short of the estimate even when estimates are unbiased; a Bayesian adjustment of estimates is proposed. Abstract-level reading only.