Skip to content
ML Academy

LLMs, GenAI, RAG, agents, evaluation · insight-0651

Encode or parameterise model output before it reaches SQL, HTML or a shell

Evidence type
documented fact
Difficulty
easy
Roles
ML engineer, Applied AI
Industries
e-commerce, saas

Summary: situation, action, reason, boundary

  1. Situation

    Generated text is inserted into a web page, a database query, a template or a command line.

  2. Action

    Treat model output like untrusted user input: use parameterised queries, context-aware encoding for HTML and other sinks, a strict content security policy for rendered output, and allow-lists for any identifiers that select tables, files or commands.

  3. Reason

    OWASP's Improper Output Handling entry describes how unvalidated LLM output can lead to XSS, SSRF, privilege escalation or remote code execution, and recommends zero-trust handling, encoding and parameterised queries.

  4. Boundary

    Any pipeline where model text is consumed by another program.

Symptom or risk

Output shaped by a user or by injected content becomes cross-site scripting, SQL injection or command execution in a downstream system.

Exceptions

Output shown only as plain escaped text to the same user who wrote the prompt, with no tools involved.

Trade-offs

Strict encoding can break legitimate rich formatting; allow a small, sanitised Markdown subset.

Pitfalls

Validating the model's input but not its output, or trusting output because "our model would not do that".

Lessons and concepts

day 53day 47
Untrusted input boundariesStructured outputs and validation

Claims and their sources