The first two questions are asked in the same breath and have almost nothing in common. One
is a contract and network-topology question with a factual answer. The other is a data
classification question your organisation has probably already answered for every other
system and has not yet mapped onto this one.
Question 01
Is our data safe? Is it being used to train the model?
Asked in the room as one question. It is three, and they have different answers.
Separate training, retention, and access.
Conflating them is why the answer sounds evasive even when it is not.
Training is whether your content is used to update model weights. This is a
contractual property of the endpoint you are calling, not a property of the model. The same
model, under the same brand name, is typically reachable through a consumer subscription, an
enterprise workspace tier, a first-party API, and one or more cloud marketplace endpoints —
and the training default is not the same across all four. Enterprise and API tiers generally
exclude training on customer content by default; consumer tiers historically have not. Read
the terms for the exact endpoint, get it in the agreement, and stop asking the question about
the vendor.
Retention is how long your content sits on someone else’s disk, and it is a
separate switch. Content can be excluded from training and still be retained for thirty days
for abuse monitoring, cached server-side for a prompt-cache TTL, logged by an intermediate
gateway, and replicated across regions. Zero-retention modes exist on most enterprise
endpoints and usually have to be requested and configured. Ask for the retention window, the
abuse-review exception, the subprocessor list, and the region the inference actually runs in.
Access is who can read it — including your own people. A workspace where every
member can see every project’s knowledge base has just flattened the entitlement model you
spent a decade building. This is usually the real exposure, and it is entirely yours to fix.
One clarification worth making publicly, because it prevents an entire category of anxiety:
a model does not learn from your conversation while you are having it. Everything the model
“knows” about your business inside a session is text placed into the request. In-context is
not training. The reason the session appears to remember is that the transcript is being
resent on every turn.
What to implement. One egress path. Route every model call through a
gateway you operate, so there is exactly one endpoint with one contract and one log. Then
block the consumer surfaces at the network and identity layer — not because they are
unsafe in principle, but because you cannot answer any of the three questions above for a
surface you do not control. Shadow usage is not a policy failure, it is a routing failure:
it happens wherever the sanctioned path is slower than the unsanctioned one.
Ask the vendor
Training exclusion, in the contract
Retention window per endpoint
Abuse-monitoring exception
Human review conditions
Inference region
Subprocessor list
Cache TTL
Question 02
What am I actually allowed to upload — client names, PII, portfolio data?
The question that has no good answer at the level it is asked.
If you cannot answer this for email, a ticketing system, or a spreadsheet on a shared drive,
you will not answer it for AI. Map your existing classification tiers onto approved surfaces
and publish the mapping — not a list of forbidden words.
A prohibited-terms list fails on contact. People cannot apply it, it produces false comfort,
and it makes the compliant path slower than the shadow path. What works is a short decision
rule: for each classification tier your firm already defines, name the surfaces it may reach
and the surfaces it may not, and make the approved surface at least as convenient. One page.
Reviewable by legal in an afternoon.
There are two twists that are genuinely new and worth raising with risk functions rather
than hand-waving:
- Aggregation. A model is extremely good at combining individually
permitted facts into a conclusion that is not permitted. Twenty disclosed data points can
constitute a restricted inference. Classification schemes designed for documents do not
describe conclusions.
- Information barriers. A chat surface has no concept of a Chinese wall.
An assistant with retrieval across both sides of a barrier is a control failure even when
every individual document was lawfully accessible to the service account that indexed it.
This is the finding that ends pilots in regulated firms, and it is an architecture
decision made months earlier.
Redaction helps and does not save you. Pattern-based detection of names, account numbers and
identifiers has a false-negative rate that makes it a speed bump rather than a control; treat
it as defence in depth on top of the real control, which is what data the surface can reach
at all. Where you do need to send sensitive fields, prefer deterministic tokenisation with
the mapping held on your side, so the response can be rehydrated locally and the vendor never
holds the join.
The rule that survives contact. Do not govern what people type. Govern
what the surface can reach, who it authenticates as, and where the output can land. A
control that depends on every employee remembering a policy under time pressure is not a
control — it is a hope with a training deck.
