Books, papers and documentation
- 811sources
- 810identity verified
- 1not verified or restricted
- 27core reading
Statistics23
ACI updates the miscoverage level online to keep long-run coverage under distribution shift; Day 26-27 online intervals.
original paperverified- original paperverifiedopen (arXiv)
Box-Cox power transformation family with likelihood-based choice of lambda; variance stabilization before modelling; Day 22.
original paperverified- original paperverifiedpaywalled (OUP)
- original paperverifiedpublisher; abstract free
Weighted conformal methods with coverage-gap bounds when exchangeability fails (drift, time series); Day 26.
original paperverifiedCQR calibrates quantile-regression intervals with conformal correction, giving adaptive-width intervals with coverage guarantees; Day 26.
original paperverified- original paperverified
- original paperverified
- original paperverifiedpublisher (paywalled); metadata free
- original paperverifiedpaywalled (Springer)
Original publication using the 442-patient diabetes data; Project 009.
original paperverified- original paperverifiedfree online
Ljung-Box portmanteau test for residual autocorrelation (white-noise check of model residuals); Day 22-23.
original paperverified- original paperverifiedpaywalled (OUP)
Basis for combining a large judge-scored set with a small human-labelled set.
original paperverifiedopen access (arXiv)Introduces the lasso (L1 penalty) that shrinks and sets some coefficients exactly to zero; Day 11 regularization, Day 17 feature selection.
original paperverifiedElastic net combines L1 and L2 penalties, handling groups of correlated predictors better than lasso alone; Day 11.
original paperverifiedIntroduces ridge (L2-penalized) regression as a biased estimator that reduces variance under multicollinearity; Day 11 regularization.
original paperverifiedSubsampling-based selection frequencies; Day 17.
original paperverifiedfree (arXiv)Theory of (strictly) proper scoring rules including CRPS, log score and interval/quantile scores; Day 26 probabilistic forecast evaluation.
original paperverifiedBlock bootstrap for dependent (time-ordered) observations; Project 004.
original paperverified- original paperverified
Machine learning29
DBSCAN: density-based clusters of arbitrary shape with explicit noise points, parameters eps and MinPts; Day 14.
original paperverified- original paperverifiedpublisher (paywalled); metadata free
SHAP: additive feature attributions based on Shapley values, unifying LIME/DeepLIFT-style methods; Day 17 local explanations.
original paperverifiedFormalizes model reliance (permutation importance) and model class reliance across the Rashomon set of near-optimal models; Day 17.
original paperverifiedOrdered boosting and ordered target statistics to avoid target leakage / prediction shift with categorical features; Day 9 target encoding, Day 12.
original paperverified- original paperverifiedfree online (open access)
- original paperverifiedpublisher (paywalled); metadata free
- original paperverifiedfree online
- original paperverifiedopen access
- original paperverifiedfree PDF on the group site
Gradient boosting as stagewise functional gradient descent on a loss; also introduces partial dependence plots; Day 12 boosting, Day 17 interpretation.
original paperverifiedAnomalies are isolated in fewer random splits, so short average path length signals an outlier; Day 14 anomaly detection.
original paperverifiedk-means++ seeding (D^2 sampling) gives an O(log k)-competitive expected guarantee and better clusters/speed than random init; Day 14.
original paperaccess requiredDefines data leakage formally and gives ways to detect and avoid it; core for validation design and feature audits.
original paperverifiedfree access (ACM DL)- original paperverifiedfree author copy
Histogram-based boosting with gradient-based one-side sampling (GOSS) and exclusive feature bundling (EFB); Day 12.
original paperverifiedICE plots show per-observation prediction curves, revealing heterogeneity/interactions that an averaged partial dependence plot hides; Day 17.
original paperverifiedLabel noise in evaluation data (days 18, 41).
original paperverifiedopen access (arXiv)- original paperverifiedpublisher (paywalled); metadata free
Introduces random forests (bagged trees with random feature subsets), out-of-bag error and permutation-style variable importance; Day 12.
original paperverified- original paperverifiedfree online
Defines the silhouette width for judging cluster cohesion vs separation and choosing k; Day 14.
original paperverifiedIntroduces SMOTE (interpolated synthetic minority examples); cite alongside van den Goorbergh 2022 for its effect on calibration; Day 13.
original paperverified- original paperverifiedfree (author copy via anthology)
- original paperverifiedfree online (repository record)
- original paperverifiedfree online
Accumulated local effects (ALE) plots avoid PDP extrapolation into unlikely regions when features are correlated; Day 17.
original paperverifiedGBM baseline before neural nets on tabular data (days 12, 35).
original paperverifiedopen access (arXiv)Regularized second-order tree boosting with sparsity-aware split finding and systems optimizations; Day 12.
original paperverified
Forecasting and time series13
Exponential smoothing as innovations state space models (ETS), enabling likelihood, AIC model selection and prediction intervals; Day 23.
original paperverifiedDescribes automatic ETS and the auto.arima stepwise order-selection algorithm; Day 23.
original paperverified- original paperverified
Conformal intervals for multi-horizon RNN forecasts with Bonferroni-style correction across horizons, assuming independent series; Day 26-27.
original paperverified- original paperverified
Croston's method: separately smooth nonzero demand sizes and inter-demand intervals for intermittent series; Day 26.
original paperverifiedBenchmark spanning many datasets, frequencies and horizons with a non-leaking pretraining split; evaluates statistical, DL and foundation models; Day 27.
original paperverifiedExtends STL to several seasonal periods by iterative decomposition; the MSTL model in statsforecast follows this idea.
original paperverifiedForecast all levels independently and reconcile via regression (optimal combination), versus bottom-up/top-down; Day 26.
original paperverifiedMinT reconciliation: minimizes trace of reconciled forecast error covariance, generalizing OLS/WLS reconciliation; Day 26 hierarchical forecasting.
original paperverifiedParadigm of maximizing sharpness subject to calibration; PIT histograms as calibration diagnostics; Day 26.
original paperverifiedSTL decomposes a series into trend, seasonal and remainder via iterated loess, with robustness weights; Day 21-22.
original paperverifiedFounding paper showing combined forecasts can beat individual ones; basis for ensembles/combination in forecasting; Day 26.
original paperverified
Causality and experiments7
- original paperverified
The method behind the CausalImpact package: a counterfactual forecast from control series via a Bayesian structural time-series model; use for marketing/intervention impact lessons.
original paperverifiedopen access (Project Euclid)- Investigating Causal Relations by Econometric Models and Cross-spectral Methods (opens in a new tab)
Defines Granger causality as predictive precedence; cite it when explaining why 'Granger-causes' does not mean causal effect.
original paperverifiedpaywalled (JSTOR; free online reading with a JSTOR account) - original paperverified
Introduces the synthetic-control estimator with placebo-based inference (California Proposition 99 case); core for single-unit policy evaluation.
original paperverifiedpaywalled- original paperverified
- original paperverified
Deep learning73
Defines the Adam update with bias-corrected first and second moment estimates; supports day 33 (optimizers).
original papercore readingverifiedopen access (arXiv)Residual connections make very deep networks trainable; supports day 36 (CNNs) and the residual-stream idea reused in Transformers (day 39).
original papercore readingverifiedMax-softmax confidence as a routing baseline (days 32, 41).
original paperverifiedopen access (arXiv)TimesFM: patched decoder-only Transformer pretrained on a large time-series corpus for zero-shot point forecasting; Day 27.
original paperverifiedContrastive learning with augmentations, a projection head and large negative sets; supports day 38 (contrastive representation learning).
original paperverifiedopen access (PMLR)PatchTST: patching plus channel independence for Transformer long-horizon forecasting; Day 27.
original paperverifiedSeeds and variance in model comparisons (days 35, 40).
original paperverifiedopen access (arXiv)- original paperverifiedopen web
Variational autoencoder, ELBO and the reparameterization trick; supports day 46 (Generative Modeling).
original paperverifiedopen access (arXiv)- original paperverifiedopen web
Batch normalization and its train/inference behaviour difference; supports day 34. The paper's own "internal covariate shift" explanation should be presented as the authors' motivation, not settled fact.
original paperverifiedopen access (PMLR)- original paperverifiedopen web
- original paperverifiedopen web
Tokenizes scaled, quantized series and trains T5-style language models for zero-shot probabilistic forecasting; Day 27.
original paperverifiedGuidance trades sample diversity for fidelity, in the spirit of low-temperature or truncated sampling, by mixing conditional and unconditional estimates. Supports insight 0573.
original paperverifiedopen access (arXiv)Perplexity is an inadequate measure for compressed LLMs; on knowledge-intensive tasks (LLM-KICK) pruning degraded performance even at modest sparsity while quantization held up better. Supports insight 0551.
original paperverifiedopen access (arXiv)- original paperverified
Learning-rate range test (days 33, 34).
original paperverifiedopen access (arXiv)Shows L2 regularization and weight decay differ for adaptive optimizers and proposes AdamW; supports days 33 and 40.
original paperverifiedopen access (arXiv)- original paperverifiedopen web
- original paperverifiedopen web
Global autoregressive RNN trained across many related series producing probabilistic (likelihood-based) forecasts; Day 27.
original paperverifiedIntroduces He (Kaiming) initialization for ReLU networks and PReLU; supports day 34 on initialization choice for ReLU-family activations.
original paperverifiedForward noising and learned reverse denoising process (noise-prediction objective); supports day 46.
original paperverifiedopen access (NeurIPS proceedings)Origin of the teacher-student (distillation) recipe; says nothing about LLM outputs, so use it for the mechanism only.
original paperverifiedMonte Carlo dropout at inference (days 34, 26).
original paperverifiedopen accessDropout as regularization and its train vs eval behaviour; supports day 34 (Training Deep Networks).
original paperverifiedopen access (JMLR)- original paperverifiedopen access
Linear probe then fine-tune (day 40).
original paperverifiedopen access (arXiv)Attention cost and memory (days 39, 42, 54).
original paperverifiedopen access (arXiv)Loss reweighting under class imbalance (days 13, 41).
original paperverifiedopen access (arXiv)Introduces the Fréchet Inception Distance (FID), comparing statistics of real and generated images in Inception feature space. Supports insight 0574.
original paperverifiedopen access (arXiv)Defines GELU, the activation used in BERT/GPT-style Transformers; supports day 32 (Activations and Losses).
original paperverifiedopen access (arXiv)Generator-discriminator minimax game; supports day 46 comparison of generative families (GAN vs VAE vs diffusion).
original paperverifiedopen access (NeurIPS proceedings)Multi-query attention (one key-value head) speeds decoder inference but can lose quality; grouped-query attention (an intermediate number of key-value heads) gets close to multi-head quality at close to multi-query speed. Supports insight 0506.
original paperverifiedopen access (ACL Anthology)Convolutional networks (LeNet) trained end-to-end with gradient descent; supports day 36 (CNNs) on weight sharing and local receptive fields.
original paperverifiedSmall-batch normalization choice (days 34, 36).
original paperverifiedopen access (arXiv)Diffusion in an autoencoder's latent space to cut compute, with cross-attention conditioning; supports day 46.
original paperverifiedEarly layers are general, later layers task-specific, and transfer degrades with task distance; supports day 40 (transfer learning / fine-tuning).
original paperverifiedopen access (NeurIPS proceedings)Deep CNN with ReLU, dropout and GPU training on ImageNet; historical anchor for day 36 (CNNs).
original paperverifiedopen access (NeurIPS proceedings)- original paperverified
Measures sample quality (precision) and coverage of the real distribution (recall) separately, which single-number metrics such as FID conflate. Supports insight 0574.
original paperverifiedopen access (NeurIPS proceedings)Per-example normalization independent of batch size, used in Transformers; supports days 34 and 39.
original paperverifiedopen access (arXiv)Formal account of vanishing/exploding gradients over long sequences; supports days 34 and 37 (why plain RNNs struggle).
original paperverifiedClassic statement of backpropagation as gradient computation through hidden layers; supports day 33 (Backpropagation and Optimization).
original paperverifiedGated memory cell for long-range dependencies; supports day 37 (Sequence Models).
original paperverifiedIn code and math (instruction fine-tuning and continued pretraining), LoRA substantially underperformed full fine-tuning on the target domain but better preserved base-model performance outside it. Supports insight 0547.
original paperverifiedopen access (arXiv)- original paperverifiedopen web
Mixed precision caveats (days 34, 54).
original paperverifiedopen access (arXiv)- original paperverifiedopen web
- original paperverifiedopen web
Deep stacks of fully connected blocks with backward/forward residual links and optional interpretable trend/seasonality bases; evaluated on M3/M4/TOURISM; Day 27.
original paperverifiedExtends N-BEATS with multi-rate input pooling and hierarchical interpolation for efficient long-horizon forecasting; Day 27.
original paperverified- original paperverifiedopen web
- original paperverifiedopen web
Normalization placement and warmup (days 34, 39).
original paperverifiedopen access (arXiv)Gradient clipping; exploding and vanishing gradients (days 34, 37).
original paperverifiedopen accessIntroduces the InfoNCE contrastive loss used by many embedding models; supports day 38.
original paperverifiedopen access (arXiv)- original paperverifiedopen web
- original paperverified
- original paperverifiedopen web
- original paperverifiedopen web
Mixture-of-experts selects different parameters for each input, giving a sparsely activated model with a very large parameter count at constant computational cost. Supports insight 0505.
original paperverifiedopen access (JMLR, CC BY 4.0)Attention model handling static covariates, known-future and observed inputs with variable selection and quantile outputs; Day 27.
original paperverifiedAdam vs SGD generalization trade-off (day 33).
original paperverifiedopen access (arXiv)Gradient (activation) checkpointing (days 34, 40).
original paperverifiedopen access (arXiv)Memorization capacity; label noise; overfitting a batch (days 34, 35, 41).
original paperverifiedopen access (arXiv)Origin of variance-preserving (Xavier/Glorot) initialization and analysis of saturating activations; supports days 32 and 34 (activations, training deep networks).
original paperverifiedopen access (PMLR)- original paperverifiedopen web
- original paperverified
Moirai: masked-encoder universal forecaster with multi-patch-size projections and any-variate attention, trained on LOTSA; Day 27.
original paperverifiedLabel smoothing trade-offs (days 32, 34).
original paperverifiedopen access (arXiv)GPU memory budgeting for training (days 40, 44, 54).
original paperverifiedopen access (arXiv)
NLP and LLMs123
Transformer with scaled dot-product and multi-head attention, positional encodings; core source for days 39 and 42.
original papercore readingverifiedopen access (NeurIPS proceedings)Encoder-only masked-language-model pretraining then fine-tuning; supports days 42-43 (encoder vs decoder architectures, pretrain/fine-tune).
original papercore readingverifiedopen access (ACL Anthology)CBOW and skip-gram word embeddings; supports day 38 (Embeddings and Representation Learning).
original papercore readingverifiedopen access (arXiv)Documents position, verbosity and self-enhancement biases of LLM judges and their agreement with humans; supports days 45 and 51 (LLM-as-judge evaluation).
original papercore readingverifiedopen access (arXiv)In-context (few-shot) learning at 175B scale; supports days 42-43 and 47 (prompting without weight updates).
original papercore readingverifiedopen access (NeurIPS proceedings)Frozen base weights plus trainable low-rank update matrices, mergeable at inference; core source for day 44 (PEFT).
original papercore readingverifiedopen access (arXiv)Accuracy drops when the relevant passage sits in the middle of a long context; supports days 48 and 51 (ordering and limiting retrieved context).
original papercore readingverifiedopen access (ACL Anthology)Additive attention removes the fixed-length bottleneck of seq2seq; supports day 39 (Attention and Transformers).
original papercore readingverifiedopen access (arXiv)Interleaves reasoning traces with tool actions and observations; core source for day 49 (agent loop).
original papercore readingverifiedopen access (arXiv)Combines a dense retriever with a seq2seq generator over a non-parametric memory; core source for day 48.
original papercore readingverifiedopen access (NeurIPS proceedings)Bi-encoder sentence embeddings for efficient semantic search vs cross-encoding every pair; supports days 38, 41 and 48.
original papercore readingverifiedopen access (ACL Anthology)SFT then reward model then PPO (RLHF) pipeline behind InstructGPT; supports day 45 (alignment).
original papercore readingverifiedopen access (arXiv)Describes how QASPER questions, answers and evidence were collected; supports project-038 dataset description.
original paperverifiedopen access (ACL Anthology)Reward improvements in RLHF came largely from longer outputs; a length-only reward reproduced most downstream gains over SFT; reward models were the main source of the length bias. Supports insight 0570.
original paperverifiedopen access (arXiv)Origin of inverse document frequency weighting; supports day 41 (TF-IDF baseline).
original paperverified- original paperverified
Introduces CLINC150 and the out-of-scope evaluation problem; supports project-049.
original paperverifiedopen access (ACL Anthology)- original paperverified
- original paperverified
Attention weights often do not track feature importance; supports day 39 caveat against reading attention maps as explanations.
original paperverifiedopen access (ACL Anthology)Nuance to Jain and Wallace (day 39).
original paperverifiedopen accessVenue not read from the page; cite as arXiv. The calibration-set robustness is the authors' claim.
original paperverifiedLinear text-classification baselines (day 41).
original paperverifiedopen accessEmbedding domain shift (days 38, 48, 51).
original paperverifiedopen access (arXiv)Capability-by-test-type matrix (minimum functionality, invariance, directional expectation tests); supports golden-set design in project-043.
original paperverifiedopen access (ACL Anthology)Documents how BillSum was assembled; supports project-046.
original paperverifiedopen access (ACL Anthology)Few-shot accuracy swings from near chance to near state of the art depending on prompt format, chosen examples and their order; names majority-label, recency and common-token biases. Supports insight 0523.
original paperverifiedopen access (PMLR)LLMs tend to be overconfident when verbalising confidence; sampling consistency and aggregation help but no single method wins consistently. Supports insight 0565.
original paperverifiedopen access (arXiv)Few-shot exemplars with intermediate reasoning steps improve multi-step tasks in large models; supports day 47 (prompt engineering).
original paperverifiedopen access (arXiv)Supports the claim that many-variant pairwise comparisons are summarised with a Bradley-Terry model and reported with confidence intervals.
original paperverifiedopen access (arXiv)Supports insight 1506.
original paperverifiedopen access (arXiv)Defines (normalized) discounted cumulated gain for graded relevance; supports day 51 (retrieval metrics such as nDCG@k).
original paperverifiedNear-duplicate detection and evaluation validity (days 41, 43).
original paperverifiedopen accessLearning a reward model from pairwise human comparisons; conceptual origin of RLHF for day 45.
original paperverifiedopen access (NeurIPS proceedings)Dual-encoder dense retrieval trained with in-batch negatives, compared with BM25; supports days 48 and 51.
original paperverifiedopen access (ACL Anthology)Optimizes preferences directly with a classification-style loss, no explicit reward model or RL loop; supports day 45.
original paperverifiedopen access (arXiv)Negative sampling and subsampling for skip-gram; supports day 38 and the contrastive-objective thread.
original paperverifiedopen access (NeurIPS proceedings)Fine-tuning examples that introduce new knowledge are learned much more slowly than examples consistent with the model's knowledge, and as they are learned the model's tendency to hallucinate increases linearly. Supports insight 0540.
original paperverifiedopen access (arXiv)Domain adaptation of pretrained encoders (days 38, 40, 43).
original paperverifiedopen accessA second phase of pretraining on in-domain text (DAPT) and on unlabelled task data (TAPT) improved downstream performance across four domains. Supports insight 0544.
original paperverifiedopen access (ACL Anthology)Introduces BANKING77 and reports few-shot intent detection with sentence encoders; supports projects 045 and 048.
original paperverifiedopen access (ACL Anthology)Each request's key-value cache is large and grows and shrinks dynamically; wasted cache memory limits batch size and throughput. Supports insight 0506.
original paperverifiedopen access (arXiv)- original paperverified
- original paperverifiedopen access (arXiv)
Binary judgments are inadequate for long-form text that mixes supported and unsupported claims; FActScore is the share of atomic facts supported by a knowledge source. Supports insight 0514.
original paperverifiedopen access (ACL Anthology, CC BY)The order of few-shot samples can make the difference between near state-of-the-art and random-guess performance; good orders do not transfer between models. Supports insight 0523.
original paperverifiedopen access (ACL Anthology, CC BY 4.0)Unsupervised fine-tuning gave some improvement, but RAG consistently outperformed it for both knowledge seen in training and entirely new knowledge. Supports insight 0540.
original paperverifiedopen access (ACL Anthology, CC BY 4.0)Fine-tuning variance across seeds (days 40, 43).
original paperverifiedopen access (arXiv)Model cascades and routing; results are task-specific and from 2023 API prices; supports day 54.
original paperverifiedopen accessEmbeddings from global co-occurrence statistics; supports day 38 comparison of count-based vs predictive embeddings.
original paperverifiedopen access (ACL Anthology)Multi-metric, multi-scenario evaluation (accuracy, calibration, robustness, fairness, efficiency); supports day 45 argument against single-number leaderboards.
original paperverifiedopen access (arXiv)Anisotropy makes raw cosine values hard to interpret (day 38).
original paperverifiedopen accessTokenizer quality affects task performance (days 41, 42).
original paperverifiedopen accessSupports insight 1512. Two snapshots of two models in 2023; not a general rate of drift.
original paperverifiedopen access (arXiv)Measures dated snapshots of a hosted service in 2023; drift size is specific to that study. Same paper as paper-chen-2023-chatgpt-behavior-drift (expansion-insights-llm-1); kept because insight-1583 cites this id.
original paperverifiedSupports insight 1502. A synthetic keyword-inclusion task, so the 68% figure is not a threshold for other tasks.
original paperverifiedopen access (arXiv)SVD of the term-document matrix as an early dense semantic representation; supports day 38 and 41 (from sparse to dense).
original paperverifiedIFEval uses about 500 prompts with 25 types of automatically checkable verifiable instructions (e.g., word counts, keyword frequency), avoiding human-evaluation cost and LLM-judge bias. Supports insight 0562.
original paperverifiedopen access (arXiv)Cosine vs dot product caveats (days 38, 48).
original paperverifiedpublisher (arXiv 2403.05440 open)DPO can find solutions that exploit out-of-distribution responses and is sensitive to shift between model outputs and preference data; carefully tuned PPO beat DPO across the dialogue and code tasks studied. Supports insight 0568.
original paperverifiedopen access (arXiv)Introduces SQuAD 2.0 and the requirement to abstain on unanswerable questions.
original paperverifiedTokenization effects on cost and context (days 41, 42, 54).
original paperverifiedopen access (arXiv)Supports insight 1522. Grade-school maths with 2022-era models only.
original paperverifiedopen access (arXiv)Chain-of-thought explanations can systematically misrepresent the true reason for a prediction; biasing features such as always placing the answer at option A sway answers but are typically not mentioned in the explanation. Supports insight 0521.
original paperverifiedopen access (arXiv)Shows that LLM judges' pairwise verdicts can flip when answer order is swapped and proposes order-balanced calibration; supports project-042.
original paperverifiedopen access (ACL Anthology)LLM relevance labelling for retrieval evaluation; supports day 51.
original paperverifiedfreeWithout external feedback, LLMs struggle to self-correct reasoning and performance sometimes degrades after self-correction. Supports insight 0516.
original paperverifiedopen access (arXiv)QA accuracy is strongly tied to the number of relevant pretraining documents; retrieval augmentation reduces that dependence. Supports insight 0504.
original paperverifiedopen access (PMLR)Introduces the RNN encoder-decoder and the gated recurrent unit (GRU); supports day 37.
original paperverifiedopen access (ACL Anthology)LLM auto-annotators prefer longer outputs; a regression-based length control made the metric harder to game by verbosity and raised correlation with human-preference rankings. Supports insight 0558.
original paperverifiedopen access (arXiv)Restricting output to structured formats such as JSON significantly lowered reasoning performance, with stricter constraints generally causing larger drops. Supports insight 0533.
original paperverifiedopen access (ACL Anthology)Fusion-in-Decoder; answer quality grows with the number of retrieved passages; supports day 48 (context assembly).
original paperverifiedopen access (ACL Anthology)A 65B LLaMA fine-tuned on 1,000 curated prompt-response pairs without RLHF performed strongly; the authors conclude almost all knowledge is learned in pretraining. Supports insight 0542.
original paperverifiedopen access (arXiv)LLM evaluators rated their own outputs higher than others' outputs that humans judged equal, and self-recognition ability correlated with self-preference. Supports insight 0559.
original paperverifiedopen access (arXiv)Supports insight 1507. Simulated conversations, not production traffic.
original paperverifiedopen access (arXiv)Supports insight 1504.
original paperverifiedopen access (arXiv)Train-test overlap in NLP benchmarks (days 15, 41).
original paperverifiedopen accessTask-specific embedding evaluation (days 38, 45, 51).
original paperverifiedopen accessByte-pair-encoding subword tokenization for open vocabularies; supports days 41-42 (tokenization and its effects on cost and behaviour).
original paperverifiedopen access (ACL Anthology)Benchmark test data leaking into pretraining corpora inflates scores; supports day 45 (evaluation validity).
original paperverifiedopen access (ACL Anthology)Large human study of intrinsic vs extrinsic hallucinations in abstractive summaries; textual-entailment measures correlated better with human faithfulness judgements than standard metrics such as ROUGE. Supports project-046.
original paperverifiedopen access (ACL Anthology)Fine-tuning instability (days 40, 43).
original paperverifiedopen access (arXiv)Bottleneck adapter layers as an early PEFT method; supports day 44 comparison of adapters vs LoRA (adapters add inference latency).
original paperverifiedopen access (PMLR)Cross-encoder re-ranking of first-stage candidates; supports day 51 (retrieve-then-rerank).
original paperverifiedopen access (arXiv)Open questions on whether LLM judgments can be trusted to evaluate retrieval systems; supports day 51.
original paperverifiedfreeIntrinsic vs extrinsic embedding evaluation (day 38).
original paperverifiedopen accessLoRA on a 4-bit quantized frozen base (NF4, double quantization, paged optimizers); supports day 44 memory trade-offs.
original paperverifiedopen access (NeurIPS proceedings)Meaning-preserving formatting changes caused up to 76 accuracy points of difference for one model; sensitivity persisted with larger models, more shots and instruction tuning; authors recommend reporting a performance range across formats. Supports insight 0525.
original paperverifiedopen access (arXiv)Near-duplicate leakage in text splits (days 41, 45).
original paperverifiedopen accessReference-free RAG metrics such as faithfulness, answer relevance and context relevance; supports day 51. LLM-computed metrics need their own validation.
original paperverifiedopen access (ACL Anthology)Fine-tuned GPT-3 on fiction in 2021; the compounding-error discussion is the authors' qualitative limitation, not a measured rate for prompted models.
original paperverifiedopen accessRandomly replacing demonstration labels barely hurt performance across 12 models; demonstrations mainly convey label space, input distribution and format. Supports insight 0524.
original paperverifiedopen access (ACL Anthology, CC BY 4.0)Almost all tested models degrade substantially as input length grows despite passing simple needle-in-a-haystack tests; only about half of models claiming 32K+ context hold up at 32K. Supports insight 0502.
original paperverifiedopen access (arXiv)Power-law relationships between loss and model size, data and compute; supports day 43, read together with Hoffmann et al. 2022, which revised the compute-optimal allocation.
original paperverifiedopen access (arXiv)Optimising a proxy reward model eventually lowers the gold reward (overoptimisation), with functional forms that depend on the method and on KL distance from the initial policy. Supports insight 0567.
original paperverifiedopen access (PMLR)Sampling diverse reasoning paths and choosing the most consistent answer beats greedy chain-of-thought decoding on arithmetic and commonsense benchmarks (e.g., GSM8K +17.9 points). Supports insight 0511.
original paperverifiedopen access (arXiv)When a model knows a fact, sampled responses tend to agree; hallucinated facts make samples diverge, giving a black-box detection signal. Supports insight 0515.
original paperverifiedopen access (ACL Anthology, CC BY)Source of the AskUbuntu similar-question benchmark and its BM25 candidate lists; supports project-047.
original paperverifiedopen access (ACL Anthology)LSTM encoder-decoder for sequence transduction with a fixed-size bottleneck vector; supports day 37 and motivates attention (day 39).
original paperverifiedopen access (NeurIPS proceedings)- original paperverified
Sentence-level NLI aggregation for summary inconsistency detection plus a benchmark of human consistency labels; supports project-046 detector choice.
original paperverifiedopen access (ACL Anthology)Explains repetitive/bland output from maximization decoding and proposes nucleus (top-p) sampling; supports day 42 (decoding, temperature, top-p).
original paperverifiedopen access (arXiv)Supports insight 1501. Priority between instruction sources is a trained, imperfect behaviour.
original paperverifiedopen access (arXiv)Guided-instruction prompting to detect instance- and partition-level contamination; supports day 45 contamination checks.
original paperverifiedopen access (arXiv)- original paperverified
Self-supervised learning of when and how to call APIs; supports day 49 (tool use).
original paperverifiedopen access (NeurIPS proceedings)Five assistants trained with human feedback consistently showed sycophancy across four free-form tasks; in human preference data, responses matching the user's views were more likely to be preferred. Supports insight 0520.
original paperverifiedopen access (arXiv)- original paperverified
Model size and training tokens should scale roughly together for compute-optimal training (Chinchilla); supports day 43.
original paperverifiedopen access (arXiv)- original paperverified
Models produced many false answers that mimic popular misconceptions, and the largest models tested were generally the least truthful. Supports insight 0519.
original paperverifiedopen access (ACL Anthology, CC BY 4.0)RLHF generalised better out of distribution than SFT but substantially reduced output diversity. Supports insight 0569.
original paperverifiedopen access (arXiv)Gradual unfreezing and layer-wise learning rates (days 40, 43).
original paperverifiedopen accessSupports insight 1503.
original paperverifiedopen access (arXiv)Across 162 roles, four model families and 2,410 factual questions, adding a persona to the system prompt did not improve performance over a no-persona control; the effect of a given persona was largely random. Supports insight 0535.
original paperverifiedopen access (ACL Anthology, CC BY 4.0)LMs struggle with less popular facts; scaling mainly helps popular knowledge; retrieval helps most on the long tail and can hurt on popular entities (Introduction); adaptive retrieval by popularity. Supports insight 0504.
original paperverifiedopen access (ACL Anthology, CC BY)- original paperverified
Failure taxonomy for multi-agent systems; its categories are used here only as a labelling aid, and findings are not claimed to carry over to single-agent tool use.
original paperverifiedopen accessArgues hallucinations persist because training and evaluation reward guessing over acknowledging uncertainty, since most benchmarks penalise uncertain answers; proposes changing benchmark scoring so abstaining is not penalised. Evolving research. Supports insight 0517.
original paperverifiedopen access (arXiv)Meta-analysis finding underpowered experiments common in NLP, including small test sets that cannot reliably detect typical improvements. Supports insight 0555.
original paperverifiedopen access (ACL Anthology)- original paperverified
- original paperverified
Agent evaluation by end state and repeated-trial consistency; supports days 49 and 50.
original paperverifiedopen access
MLOps and engineering3
Names ML-specific debt such as entanglement, hidden feedback loops, glue code and configuration debt; core reading for MLOps lessons.
original paperverifiedopen accessPreprint; capacity gains are the authors' measurements on specific models and GPUs.
original paperverifiedA concrete checklist of tests for data, model, infrastructure and monitoring; use it as a production-readiness rubric in projects.
original paperverifiedopen access (PDF on Google Research)
Security and governance12
Instructions planted in retrieved content can hijack LLM-integrated apps; core source for day 53 and RAG/agent threat models (days 48-49).
original papercore readingverifiedSupports reporting attack success together with the attacker's sample budget. Exact numbers are model versions of late 2024 only.
original paperverifiedopen access (arXiv)Architectural defences against prompt injection with utility trade-offs; supports days 49 and 53.
original paperverifiedopen accessDiffusion models memorised individual training images and emitted them at generation time; in the authors' tests they were much less private than earlier generative models such as GANs. Supports insight 0575.
original paperverifiedopen access (USENIX)Memorized training sequences, including personal data, can be extracted from a deployed LM; supports day 53 (privacy risk of training and fine-tuning data).
original paperverifiedopen access (USENIX)Fine-tuning on a handful of adversarial examples broke safety guardrails of an aligned model; fine-tuning on benign, commonly used datasets also degraded safety, to a lesser degree. Supports insight 0543.
original paperverifiedopen access (arXiv)Attack-success numbers are only comparable under a fixed attack set, behaviours and judge.
original paperverifiedopen access (arXiv)Goal hijacking and prompt leaking via direct prompt injection; supports day 53.
original paperverifiedopen access (arXiv)Knowledge-base poisoning: a few planted passages can steer RAG answers; supports write-control and provenance on the corpus.
original paperverifiedopen access (arXiv)- original paperverifiedopen access (arXiv)
Jailbreak strings can be found by search and may transfer; a fixed list of known jailbreaks is not a bound on attacker capability.
original paperverifiedopen access (arXiv)Models emit non-existent package names; an attacker can register such a name. Supports allow-listing installs.
original paperverifiedopen access (arXiv)
General2
Goodhart's law for KPIs and model objectives; preprint, not peer reviewed.
original paperverified- original paperverifiedfree, CIA historical review release