Training-data risks

Beyond what you send at inference, there's a subtler class of risk in the data that shapes models — training sets, fine-tuning data, and the corpora that ground RAG. These are the risks that turn a data-handling mistake into a permanent, embedded one.

Memorization & extraction

Models can memorize verbatim snippets of their training data, and attackers can sometimes extract them with the right prompts — a real concern if you fine-tune on sensitive data. If you fine-tune a model on customer records, that PII is now latent in the model's weights and may surface in unrelated conversations, for unrelated users. The mitigation is upstream: don't fine-tune on raw sensitive data. Redact or synthesize it first; assume anything in the training set can eventually come out.

Data poisoning (the supply-chain angle)

• Training/fine-tuning poisoning: an attacker who influences your training data can plant backdoors or biases. Relevant if you accept user data into a training pipeline — treat that pipeline as an attack surface, not a convenience.
• RAG poisoning: the more immediate version for most builders. If your retrieval corpus includes content outsiders can influence (product reviews, submitted docs, scraped pages), an attacker plants malicious or false content that your system later retrieves and presents as grounded truth. This is indirect injection and misinformation, entering through the corpus.
• Defense: provenance and trust levels on corpus content — know what's authoritative vs. user-generated, and treat user-generated retrieved content as untrusted (fence it, don't let it drive actions). Curate what enters the index.

Safe-data practices

• Prefer synthetic or masked data for fine-tuning, testing, and demos. Realistic data that preserves shape without exposing real people is the safe default (synthetic-data generation is a real product category, because the need is universal).
• Segregate and label corpus content by trust level; never let untrusted retrieved content take actions or override instructions.
• Apply retention and deletion to embeddings too: a customer's right-to-be-forgotten must reach the vector store and any fine-tune, not just the primary database. Embeddings are derived personal data.
• Vet third-party models and datasets — a model or dataset from an untrusted source is a supply-chain risk (poisoned weights, license traps). Provenance matters for models as much as for dependencies.