Erik S., Barry Zhang (Anthropic) · 2024
Vendor engineering guidance (practitioner experience, not a controlled study); supports day 49 workflow-vs-agent choice.
Checked 2026-10-11: Page title, date 2024-12-19, authors as bylined. Content: workflows use predefined code paths, agents direct their own tool use; start with the simplest solution and add agentic complexity only when needed; agents trade latency and cost for performance and risk compounding errors; recommends sandboxed testing, guardrails and stopping conditions such as a maximum number of iterations; tool definitions deserve the same care as a human interface.
Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, Jeremy Hadfield (Anthropic) · 2025
Vendor engineering guidance (practitioner experience); supports context and memory management in long agent runs.
Checked 2026-10-12: Opened 2026-10-12. Title, bylined authors, date 2025-09-29. Content: compaction summarizes a conversation near the context limit and continues in a fresh window, keeping key decisions and open issues and dropping redundant output; tool result clearing removes raw tool outputs deep in history; structured note-taking persists notes outside the context window; just-in-time retrieval keeps lightweight references and loads data on demand; sub-agents return condensed summaries.
Justin Young (Anthropic) · 2025
Vendor engineering report (one team's experience, not a controlled study); supports long-running task design and completion verification.
Checked 2026-10-12: Opened 2026-10-12. Title, byline and date 2025-11-26 as shown. Content: agents that tried to do everything in one pass ran out of context mid-feature, and later sessions sometimes saw partial progress and declared the project finished; an initializer session writes a JSON feature list with every feature failing plus a progress file and git history; later coding agents may only change a feature's pass/fail status; one feature per session; agents often marked features done after only unit tests or curl checks, and prompting for end-to-end browser testing improved results.
Anthropic · 2024
Vendor-reported, not peer reviewed; use as a hypothesis to test on your own golden set (day 48).
Checked 2026-10-11: Page title and date 2024-09-19. Method: prepend model-generated, chunk-specific context (about 50-100 tokens) to each chunk before embedding and BM25 indexing. Vendor-reported top-20 retrieval failure rate (1 - recall@20) on their test sets fell from 5.7% to 3.7% (contextual embeddings), 2.9% (plus contextual BM25) and 1.9% (plus reranking).
Clementine Fourrier, Nathan Habib, Julien Launay, Thomas Wolf (Hugging Face) · 2023
2023 evidence about one benchmark and three codebases; current harness versions may behave differently. Vendor-adjacent (platform operator) but the numbers are reproducible from the code named.
Checked 2026-10-12: Page opened twice 2026-10-12. Title, four bylined authors, date June 23, 2023. The post runs MMLU through three implementations (original Berkeley code, HELM, EleutherAI LM Evaluation Harness at a January 2023 commit) and prints a table, for example llama-65b 0.637 HELM / 0.488 Harness / 0.636 Original, and falcon-40b 0.571 / 0.527 / 0.558, llama-30b 0.583 / 0.457 / 0.584. It attributes differences to prompt wording and to how the answer is scored (original: probabilities of the four letters; HELM: the highest-probability next token matched against the answer; early harness: log-likelihood of each full answer text). It says the harness was updated to match the original and the leaderboard would be re-run.
Ken Aizawa and Anthropic colleagues · 2025
Vendor engineering guidance (practitioner experience).
Checked 2026-10-12: Opened 2026-10-12. Title, lead author Ken Aizawa, date 2025-09-11. Content: more tools do not always help and overlapping tools can distract agents; consolidate related operations into one tool; error responses should say specifically what to fix instead of opaque codes; namespace related tools; paginate, filter or truncate large responses with sensible defaults.