Log full prompts and responses only where debugging quality genuinely requires them, behind field-level redaction, tiered sampling, and short default retention — then widen access and time only after legal signs off. Everything else gets hashed, truncated, or dropped. The policy, not the pipeline, is what survives an audit.
Quick Answer: Don't choose between "log everything" and "log nothing." Classify fields by sensitivity and debugging value, redact PII at write-time, sample raw I/O instead of storing it all, set a short default retention window, and gate raw-text access behind role and purpose. Document it as a versioned policy, not a scattered set of engineering defaults.
Why LLM Logs Break the Old Logging Playbook
Traditional application logs record structured events — a status code, a user ID, a latency number. LLM logs record unbounded natural-language payloads that can contain anything a user typed or an assistant fetched, including health details, financial data, or a competitor's leaked pricing sheet. That difference changes who needs to sign off on your logging design.
A checkout API log rarely captures a customer's medical history in the middle of a request body. An LLM chat log routinely does, because the whole point of the interface is "type anything." The three dashboards every AI product needs on day one all draw from this same log stream — quality, cost, and safety dashboards are only as good as the fields underneath them, which is exactly why the schema decision below is not a backend implementation detail you can defer.
The Debugging Case for Raw I/O
Full prompts and responses are often the only way to root-cause a quality regression. When an eval score drops, you need to see what the model actually received — including retrieved context, system prompt version, and tool outputs — not just a pass/fail label. Our companion piece on quality drift with no stack trace covers why LLM failures rarely throw exceptions; the prompt/response pair is frequently the closest thing you get to one.
The Liability Case Against It
The same payload is a PII magnet and a cost center. Every stored prompt is a potential subject-access-request target, a breach-notification trigger, and a line item in your data-warehouse bill. Regulators including the NIST AI Risk Management Framework and the EU's GDPR (via data minimization, Article 5(1)(c)) treat "we logged it because it might be useful someday" as an insufficient justification for retention. The tension isn't a gap in your engineering — it's structural, and it needs a policy, not a workaround.
A Decision Framework: Four Levers, Not One Switch
Treat LLM logging as four independent levers — redaction, sampling, retention, and access — rather than a single "log verbosity" setting. Tuning each lever separately lets you keep high debugging value on the fields that need it while cutting exposure everywhere else. Most teams collapse these into one on/off decision and regret it during their first privacy review.
- Redaction — what gets scrubbed or tokenized before storage, and at what layer.
- Sampling — what fraction of raw traffic gets full-fidelity capture versus a summary.
- Retention — how long each field class lives before automatic deletion.
- Access control — who can query raw text, under what approval, logged how.
Lever 1: Redaction and PII Handling
PII redaction in LLM logs means detecting and masking sensitive spans — names, emails, phone numbers, national IDs, health terms — before the payload is written to durable storage, not after. Doing it at write-time, not query-time, is what actually reduces your exposure surface, because a query-time filter still leaves raw PII sitting in the underlying store.
- Pattern-based redaction (regex,
Presidio-style detectors) catches structured PII fast — emails, phone numbers, credit cards — at low compute cost but misses free-text disclosures like "my daughter has type 1 diabetes." - Model-based redaction (a lightweight classifier or a smaller LLM pass) catches contextual PII pattern-matching misses, at added latency and its own error rate.
- Tokenization/pseudonymization replaces a detected span with a reversible token keyed to a separate, access-controlled vault — useful when support teams occasionally need to reconstruct a specific case.
- Hard truncation caps free-text fields at a character limit, trading debuggability for a hard privacy ceiling on worst-case payloads.
None of these are complete alone. A practical policy layers pattern-based redaction as the default pass, model-based redaction on a sampled subset for high-risk surfaces (health, finance, legal), and tokenization wherever a human will plausibly need to look something up later.
Redaction that runs only in a downstream analytics job, after the raw payload has already landed in a primary log store, doesn't reduce your breach surface — it reduces your dashboard's surface. The unredacted copy still exists somewhere.
Lever 2: Sampling Strategy
Sampling means capturing full-fidelity raw I/O for only a defined percentage or trigger-set of requests, while every other request gets a lighter-weight record. This is the single biggest lever for controlling both storage cost and privacy exposure, because most requests don't need forensic-level detail — only the ones tied to a quality signal do.
| Sampling approach | What it captures | Best for |
|---|---|---|
| Random sample (e.g. 1-5%) | Full raw prompt + response, unbiased | Baseline quality monitoring, drift detection |
| Signal-triggered capture | Full raw I/O on low confidence score, user thumbs-down, or eval failure | Debugging specific incidents |
| Always-on for new features | Full raw I/O during a defined rollout window | New prompt/feature launches needing close observation |
| Summary-only (default) | Redacted excerpt, token counts, latency, model version | Everything else |
Once an incident is closed from a failure back to an eval case, you generally don't need to keep re-capturing full raw I/O for that same failure mode — downgrade it back to summary-only sampling and let the eval suite do the ongoing monitoring instead.
Lever 3: Retention Windows
Retention is how long each field class survives before automatic, enforced deletion — and it should differ by field, not apply as one blanket TTL to the whole log row. A single retention period across every field either keeps low-risk telemetry too briefly to be useful, or keeps raw PII far longer than any debugging session actually needs.
| Field class | Suggested default retention | Rationale |
|---|---|---|
| Redacted prompt/response text | 30-90 days | Enough to catch delayed quality regressions |
| Raw (unredacted) prompt/response | 7-14 days, sampled traffic only | Debugging window for acute incidents |
| Structured metadata (latency, model, cost) | 12-24 months | Low sensitivity, high longitudinal value |
| Tokenized PII vault entries | Tied to account lifecycle, not log lifecycle | Deletion must cascade on account deletion / erasure request |
These are starting points, not prescriptions — regulated industries (health, finance) will need shorter raw-text windows, and HIPAA-adjacent contexts often push you toward avoiding raw clinical text in logs entirely rather than tuning the retention number.
Lever 4: Access Control
Access control determines who can query raw, unredacted logs, under what justification, with what audit trail. Most quality-debugging work only needs redacted logs; raw access should be the exception, granted per-incident rather than as a standing permission.
- Role-gated access: raw-text query rights limited to on-call engineers and a named data-protection contact, not the whole product org.
- Purpose-logged access: every raw-log query is itself logged with a stated reason (incident ID, ticket link) — the access log becomes evidence for your next audit.
- Time-boxed elevation: raw access grants expire automatically (e.g. 48 hours tied to an incident), rather than persisting as a standing role.
- Aggregate-first defaults: dashboards and evals default to redacted/aggregated views; raw drill-down is a deliberate, logged action, not the default query surface.
A Concrete Field-Level Logging Schema
A field-level schema means deciding sensitivity, redaction rule, and retention per field in the log row — not per table. This is the artifact that actually gets reviewed by legal and security, so it's worth writing down explicitly rather than inferring from code.
| Field | Sensitivity | Redaction rule | Default retention | Sampled or always |
|---|---|---|---|---|
request_id, session_id | Low | None | 24 months | Always |
model_version, prompt_template_id | Low | None | 24 months | Always |
user_id | Medium | Hashed (one-way) | 24 months | Always |
raw_prompt_text | High | Pattern + model redaction | 7-14 days | Sampled (1-5%) or trigger-based |
raw_response_text | High | Pattern + model redaction | 7-14 days | Sampled (1-5%) or trigger-based |
redacted_prompt_excerpt | Medium | Redacted at write time | 30-90 days | Always |
retrieved_context_ids | Medium | Reference IDs, not content | 90 days | Always |
tool_call_payloads | High | Redacted, tokenized if reconstructable | 7-14 days | Sampled |
eval_score, confidence, thumbs_feedback | Low | None | 24 months | Always |
latency_ms, token_count, cost_usd | Low | None | 24 months | Always |
Every row in this table answers a different question a reviewer will ask: what does it contain, how sensitive is it, what happens to it before storage, and how long does it live. That's the shape legal review actually wants — not a narrative description of "how our logging works."
Building the Policy Into Your Workflow, Not Just Your Pipeline
A logging policy that lives only in engineering configuration drifts the moment a new feature ships, because nobody remembers to update it and nobody's job is to check. Treating retention rules as a PRD artifact — reviewed, commented on, and versioned the same way a feature spec is — keeps the policy current and gives privacy sign-off a paper trail instead of relying on institutional memory.
This is one of the places Prodinja's Spec Studio is designed to help: it can hold your logging and retention policy as a living section of the PRD, with inline comments and readiness gates, so privacy sign-off is a tracked state in the document rather than a hallway conversation someone half-remembers six months later. The policy ships versioned alongside the feature it governs, not as a separate compliance doc nobody opens.
Framing this inside product discovery also connects back to why you're logging in the first place — understanding the underlying user need, the way Jobs to Be Done reframes a feature request as a job the user is hiring your product to do, or how a customer journey map surfaces exactly where debugging visibility matters most to the user experience. Logging policy isn't just a compliance checkbox — it's the connective tissue between what you can observe and what you're allowed to keep observing.
Common Failure Modes to Avoid
Most logging incidents trace back to a small set of repeated mistakes, not novel edge cases. Knowing the pattern in advance is cheaper than discovering it during an incident review.
- One retention period for the whole row. Forces a choice between deleting low-risk telemetry too early or keeping raw PII too long — split retention per field, as above.
- Redaction as a downstream job, not a write-time gate. The unredacted copy still exists in the primary store even after a "clean" dashboard is built on top of it.
- Standing raw-access roles instead of time-boxed grants. A permission nobody revokes is a permission that eventually gets misused or simply forgotten in an audit.
- No sampling strategy, so everything is captured "just in case." Drives up storage cost without proportionally increasing debugging value — most requests were never going to be investigated.
- Policy that exists only in code comments. Nothing for legal or security to review, approve, or point to when a regulator asks how the decision was made.
For a broader view of how logging fits into the rest of your observability stack — traces, evals, dashboards — see the complete guide to LLMOps observability, which covers where I/O logging sits relative to eval pipelines and cost tracking.
Key Takeaways
- Treat logging as four separate levers — redaction, sampling, retention, access control — rather than one on/off decision; tuning them independently preserves debugging value while cutting exposure.
- Redact at write-time, not query-time — a downstream filter doesn't reduce your actual breach surface if the raw payload already landed upstream.
- Sample raw I/O instead of capturing it universally — random baseline sampling plus signal-triggered capture (low confidence, thumbs-down, eval failure) covers most debugging needs at a fraction of the storage and risk.
- Set retention per field class, not per table — raw text should live for days, redacted excerpts for weeks to months, low-sensitivity metadata for years.
- Gate raw-text access behind role, purpose, and time — access should be the exception, logged with a reason, and auto-expiring.
- Write the schema down at the field level — sensitivity, redaction rule, and retention per field is the artifact legal review actually wants to see.
- Version the policy alongside the feature it governs — a living PRD section with sign-off tracked beats a compliance doc nobody remembers to open.
Frequently Asked Questions
What's the best way to redact PII from LLM logs without losing debugging value?
Layer pattern-based redaction (regex/Presidio-style detectors) as your default pass, add model-based redaction on a sampled subset of high-risk traffic, and use reversible tokenization for spans a support team might genuinely need to reconstruct later. This keeps most debugging context intact while removing directly identifying data from the stored payload.
How long should you retain raw LLM prompts and responses?
Most teams land on 7-14 days for raw, unredacted text, sampled rather than universal, versus 30-90 days for redacted excerpts and 12-24 months for low-sensitivity metadata like latency and token counts. Regulated domains (health, finance) should default shorter and consider avoiding raw sensitive text in logs entirely.
Do you need to log every LLM request in full to catch quality regressions?
No — a random baseline sample (roughly 1-5%) combined with signal-triggered capture on low confidence scores, negative user feedback, or eval failures catches the incidents worth investigating without the cost and risk of logging every request at full fidelity.
Who should have access to raw, unredacted LLM logs?
Access should be role-gated to on-call engineers and a named data-protection contact, granted per-incident with a logged justification, and time-boxed so elevated access expires automatically rather than persisting as a standing permission.
Is prompt and response logging a GDPR or compliance risk?
Yes, if prompts or responses contain personal data and you retain them indefinitely without a stated purpose — data minimization requirements under frameworks like GDPR expect a defined retention period and access justification, which is exactly what a field-level logging schema and redaction policy are designed to demonstrate.