The FDA vs. the black box: can you reconstruct an AI decision from 18 months ago?

The auditor asks about your ML model

Here is a scenario that keeps QA managers awake at night: an FDA inspector walks into your plant. They point to a specific batch of tablets released 18 months ago — a process where the final decision was supported by a machine-learning model.

The inspector sits down and asks four deceptively simple questions:

1. What exact input data did the model see at that precise millisecond?
2. Which specific version of the model made this prediction?
3. What was the algorithm’s confidence level (probability score)?
4. Who was the human operator who ultimately signed off on the recommendation?

If the room goes dead silent, you don’t have a technology problem. You have a fundamental problem with your AI system’s documentation architecture.

Audit trail vs. data lineage: two entirely different instruments

These two concepts are routinely confused, lumped together, or used interchangeably. The easiest way to remember the difference is by looking at the core question each one answers:

• Audit trail asks: “What happened to this data over time?” It is a chronological, immutable, un-editable log of activities: WHO did WHAT, WHEN, and WHY.
• Data lineage asks: “Where did this result come from, and what did it go through?” It is the lifecycle map documenting the origin and transformation of data, from the raw sensor measurement to its current form.

In a regulated GxP environment, you don’t choose between them — you need both. Together, they create the digital black box required to reconstruct the history of any result and any automated decision.

The anatomy of a compliant audit trail (and what’s usually missing)

To satisfy 21 CFR Part 11, a complete audit trail must capture a unique event ID, a timestamp precise to the second, the user’s system identity (login ID, not a display name), the type of event, the old vs. new value, and a documented reason for the change.

While that sounds standard, systems frequently fail audits on two specific architectural details:

• The fallacy of admin overrides. An audit trail must be technically impossible to modify or clear by anyone — including system administrators. If an IT admin can “archive old logs to improve database performance,” the system immediately fails compliance. During validation, you must explicitly try to hack or delete your own logs to prove it’s impossible.
• Ignoring clock drift (NTP server). If the clocks on your LIMS, CDS and MES are misaligned by even 30 seconds, reliable forensic reconstruction becomes impossible. Did the analyst modify the peak integration parameter before or after the supervisor signed off on the batch? Without Network Time Protocol (NTP) synchronisation across all systems, you are just guessing. NTP verification must be a standard part of your computer systems validation (CSV).

Log review is mandatory. Can AI help monitor your data integrity?

FDA inspectors no longer just ask, “Do you have an audit trail?” They ask, “How often do you review it, who does it, and what patterns of misconduct are you actively monitoring?” Failing to perform regular, documented audit trail reviews is an automatic inspection finding.

What should you be flagging automatically?

• Any modification of values after a batch has been officially closed or approved.
• Behavioural anomalies — for example, an analyst changing 150 peak integration parameters in 45 minutes. Is it legitimate re-integration, or frantic “peak-shaving” to force a result into specification?
• System activity occurring deep outside standard working hours.

Ironically, you can use unsupervised ML models such as Isolation Forest or autoencoders to monitor these logs. They learn the baseline of normal user activity and flag statistical outliers. Not every anomaly is fraud, but every anomaly deserves an explanation. Just remember: the ML model monitoring your data integrity is itself a GxP system that requires validation.

Extended data lineage: the technical passport for AI

In a traditional lab, data lineage is simple: “This number came from LIMS.” In the world of AI, that is nowhere near enough. A robust lineage pipeline must capture the instrument ID, its calibration history at that moment, the CDS software version, the integration parameters, and the reference standard lot numbers.

When an AI system supports or executes a regulatory decision, you must expand this lineage to include the four pillars of ML documentation:

1. Model version and cryptographic hash. Record the exact version number, training date, and the checksum hash of the model weights. Models evolve, and a version deployed today might behave differently than the one running a year ago.
2. Input features snapshot. Freeze the exact state of the features the model processed at that microsecond. Do not just link back to the source database — records in LIMS or MES can undergo justified, post-hoc modifications, which will distort your historical context.
3. Probability score. Saving the binary prediction (e.g. “Batch OK”) is insufficient. Archive the exact confidence metric generated by the algorithm (e.g. confidence: 94.2%).
4. SHAP/LIME explanations. For critical decisions, capture the mathematical explanation of the black box: which process variables had the highest impact on the prediction, and by how much they shifted the baseline — expressed in physical engineering units an engineer can actually interpret.

Without these four elements, your AI decision history is non-compliant. Answering the inspector’s question from the beginning of this article requires exactly this infrastructure.

How do you document algorithm decisions in your plant? Does your data lineage stop at the final number, or can you pull up the exact state of a model’s brain from a year and a half ago?

Originally published on LinkedIn on 8 July 2026.