Can You Legally Build It? The Privacy and Compliance Feasibility Check for AI Features

No single team can tell you whether you're legally allowed to build an AI feature — the answer depends on how the underlying data was collected, what it was consented for, where it gets processed, and what your model vendor's contract actually permits. Run a structured privacy and compliance check before you spec the feature, not after legal flags it in review.

Quick Answer: Having the data and having the right to use it for AI are different questions. Check consent scope and purpose limitation, separate your PII rules for training versus inference, confirm data residency and vendor data-use terms, and plan for explainability obligations — as part of the spec, before a build gets far enough that killing it is expensive.

"We Have the Data" vs. "We May Use the Data": The Question That Changes Everything

Data existing in your systems and data being legally usable for a specific AI purpose are two separate facts, answered by two separate questions. The first is an inventory question — a SELECT COUNT(*) away from an answer. The second is a purpose-limitation question, and it's where AI features quietly die in legal review after months of engineering work.

Here's the pattern that catches most teams. Your product has years of customer support messages. They're real, they're rich, they're exactly the training signal a support-triage or auto-response feature needs. Someone on the team says "we have the data," and a roadmap slot gets filled.

What that sentence skips: those messages were collected under a specific, stated purpose — "so we can help you with your issue" — inside a specific moment in the customer journey. Nobody asked, at collection time, "can we also use this to train a machine learning model that other customers' interactions will run through?" Under most privacy regimes, that's not a technicality. It's a different processing purpose requiring its own legal basis.

This is exactly the gap a structured AI feasibility check is supposed to catch before technical feasibility work even starts — because a model that would work perfectly on data you're not allowed to use for that purpose isn't feasible at all, it's just unbuilt. The fix usually isn't "abandon the feature." It's one of several reroutes:

• Re-consent a subset of users specifically for model training, accepting a smaller dataset.
• De-identify aggressively enough that the training data no longer counts as personal data under your applicable law.
• Switch approaches — use retrieval-augmented generation over live, purpose-matched data instead of fine-tuning a model on historical messages.
• Narrow the job the feature does so it maps back to a purpose customers actually agreed to — the same discipline a jobs-to-be-done framing forces you into anyway, since "the job we're hired to do" and "the job we're allowed to use this data for" need to be the same job.

None of these are free. All of them are cheaper than finding out after launch.

Consent and Purpose Limitation: Check This First

GDPR Article 5(1)(b) requires that personal data be collected for "specified, explicit and legitimate purposes" and not "further processed in a manner incompatible" with them — this single clause is the reason "we have the data" and "we may use the data for this" diverge. Before anything else in your feasibility check, identify the legal basis your original data collection relied on and whether it stretches to cover AI processing.

Under GDPR Article 6, there are six possible lawful bases, but in practice AI features usually lean on one of three:

A balancing test for legitimate interest has to weigh the user's reasonable expectations at collection time. A customer who typed a support message expected a human or a scoped automated reply — not that their words would become permanent training signal for a model serving other customers. That expectation gap is precisely what regulators and plaintiffs' attorneys have gone after.

The enforcement record backs this up. The U.S. FTC has repeatedly ordered "algorithmic disgorgement" — not just data deletion, but deletion of any model or algorithm trained on improperly obtained data — in cases including Everalbum (facial recognition trained on photos without adequate consent) and Weight Watchers' Kurbo app (children's data used beyond its stated purpose). The remedy in both cases wasn't a fine you could budget around; it was the model itself.

Purpose limitation isn't a one-time check. Re-verify it whenever a feature reuses existing data for a new job, not just when you first collect data.

PII in Training vs. Inference: Two Different Risk Profiles

Personal data that touches your model at training time and personal data that touches it at inference time carry different, non-interchangeable risks, and treating them as one problem is how teams under-scope their mitigation plan. Training risk is about permanence; inference risk is about exposure surface.

Once personal data is baked into model weights through fine-tuning, it's extremely difficult to cleanly remove without retraining — which puts training data directly in tension with a data subject's right to erasure. Inference-time data, by contrast, passes through the model per request and its risk is largely governed by logging, retention, and who else can see that request.

A feature that only ever sends data to a model at inference time — no fine-tuning, no training corpus — has a materially smaller compliance surface than one that trains on your users' data. This is a real design lever, not just a legal footnote: choosing retrieval over fine-tuning, or a general-purpose model over a custom one, can be the difference between a six-week legal review and a six-month one.

Before committing to either path, run the same data through a proper data readiness audit — sensitivity classification belongs in that audit, not as an afterthought once the model choice is already locked in.

Data Residency: Where the Model Runs Is Its Own Compliance Decision

Where your data physically gets processed and stored is a separate constraint from what you're allowed to do with it, and an AI feature can fail on residency grounds even after consent and purpose limitation are both clean. This matters most the moment a feature calls a model hosted outside the jurisdiction the data came from.

The landmark event here is the Court of Justice of the EU's Schrems II ruling (2020), which invalidated the EU-US Privacy Shield framework and put every EU-to-US data transfer arrangement under fresh scrutiny — including transfers to AI vendors whose infrastructure defaults to US regions. Cross-border transfers now generally need one of:

1. An adequacy decision (the EU has recognized a small set of countries as offering equivalent protection).
2. Standard Contractual Clauses (SCCs), paired with a documented transfer impact assessment.
3. In-region processing — routing the AI workload through a regional endpoint so the data never leaves the jurisdiction.

Residency rules aren't only an EU concern. India's data protection framework, China's PIPL, and sector rules like HIPAA for U.S. health data each impose their own localization or handling constraints, and a global product can face several simultaneously. A model endpoint that's perfectly compliant for one customer segment can be a blocker for another using the exact same feature.

Regional routing has a technical cost too — a lower-latency default endpoint often isn't the compliant one, and picking a compliant region can move you outside the response-time envelope you'd otherwise plan around when defining a latency budget for the feature. Residency and performance constraints need to be resolved together, not sequentially.

Vendor Data-Use Terms: Read the Contract Before You Build

Assuming your model vendor's default data-handling posture matches what you need is one of the most common — and most avoidable — feasibility failures, because vendor policies genuinely differ and change over time. Confirm the actual terms in writing before architecture decisions lock you in.

The questions worth asking every vendor, every time, and re-confirming at renewal:

• Does this vendor train on API/enterprise data by default, or only with explicit opt-in?
• What's the log retention window, and can it be shortened or zeroed out contractually?
• Are there subprocessors (a vendor's own downstream vendors) with different terms than the primary contract?
• Does the agreement grant you audit rights or just a compliance attestation?
• What happens to data already processed if you terminate the contract?

These aren't static facts you check once — vendor policies change with new product tiers, new regions, and new regulatory pressure, so re-verify at every material contract renewal or model upgrade. This is also exactly the kind of constraint that needs to reach engineering before implementation, which is one reason it belongs among the questions engineering needs answered before writing an AI spec — a vendor's training default determines what your own downstream promises to users can safely say.

The Right to Explanation — and Why This Belongs in the Spec, Not Legal Review

GDPR Article 22 gives individuals the right not to be subject to a decision "based solely on automated processing" that produces legal or similarly significant effects, and regulatory guidance (from the former Article 29 Working Party) interprets this as requiring meaningful information about the logic involved — not full algorithmic transparency, but a real explanation a human can act on. If your feature scores, ranks, approves, denies, or prioritizes people, this check applies whether or not you labeled it "AI."

Sectors with existing explainability norms — credit decisions, hiring, insurance underwriting — already have stricter versions of this obligation layered on top of GDPR. The NIST AI Risk Management Framework (AI RMF 1.0) treats explainability and human oversight as one of its core trustworthiness characteristics for exactly this reason: a model that can't be explained can't be meaningfully contested, appealed, or trusted by the people it affects.

The practical failure mode is architectural, not legal. Teams pick a model architecture, ship it, and only then get asked by legal or a regulator "how does this decision get made?" — at which point retrofitting an explanation layer onto a black-box model is far harder than designing for one from the start.

This is precisely the gap a feasibility check should close earlier in the process. Prodinja's Feature-to-Feasibility tool is built around exactly that idea in its current prototype form: it prompts you to characterize a feature's data needs — including sensitivity level — as part of writing the spec, so privacy, consent, and explainability constraints surface while you're still deciding what to build, rather than showing up as a blocker once legal reviews a finished feature.

Key Takeaways

• "We have the data" and "we may use the data for this" are separate questions — the first is an inventory fact, the second is a purpose-limitation question under laws like GDPR.
• Purpose limitation isn't a formality — data collected for one stated purpose (like customer support) doesn't automatically license a different purpose (like model training) without a fresh legal basis.
• Training-time PII and inference-time PII carry different risks — training bakes data permanently into weights and complicates erasure rights; inference risk is about logging, retention, and access control.
• Data residency is a separate constraint from consent — a feature can be fully consented and still blocked by cross-border transfer rules like those following Schrems II.
• Vendor data-use terms vary and change — confirm training defaults, log retention, subprocessors, and audit rights in writing, and re-check at every renewal or model upgrade.
• The right to explanation needs architectural planning, not retrofitting — if a feature scores, ranks, or decides, design for explainability before you pick a model, not after legal asks how it works.
• Algorithmic disgorgement is a real regulatory remedy — the FTC's Everalbum and Weight Watchers/Kurbo cases show regulators will order deletion of the trained model itself, not just the data.