LLMs, GenAI, RAG, agents, evaluation · insight-1611

Install only allow-listed packages when a model suggests code dependencies

Situation
A coding assistant or code-writing agent proposes imports, and a developer or pipeline runs the install.
Action
Resolve suggested dependencies against an internal mirror or allow-list with pinned hashes. Send names that are new, rarely downloaded or absent from the lockfile to human review, and run first installs in an isolated build without credentials.
Reason
Spracklen and colleagues measured hallucinated package names in 576,000 code samples from 16 models, at least 5.2% on average for commercial and 21.7% for open-source models, so non-existent names recur at scale.
Boundary
Code generation that can reach a package manager, including notebook and CI agents.

Symptom or risk

The suggested package does not exist; an attacker who registered that name on a public index now runs code on your build machine.

Disposable sandboxes with no network access to internal systems and no secrets.

New libraries need an approval step, which slows experiments.

Checking only that the name exists on the public index, which is exactly what the attacker arranges.

Spracklen et al. report average hallucinated-package rates of at least 5.2% for commercial and 21.7% for open-source code models over 576,000 samples from 16 models, with 205,474 unique hallucinated names.
We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs