We Audited an AI System Serving 100K+ Users — Here Are the 5 Critical Risks We Found

By Jarvis Inc · May 2026 · 8 min read

We recently finished auditing an AI system that makes critical decisions for over 100,000 users. The company had monitoring on latency and uptime — standard stuff. But zero visibility into model behavior.

What we found should concern anyone running ML in production.

1. Data Drift — The Model Was Living in 2021

CRITICAL

The model was trained on 2021 data. It's now 2026. Five years of behavioral shifts, market changes, and distribution drift — and the model was still predicting based on patterns that no longer exist.

Result: accuracy degrading roughly 2% per month, compounding silently. No alerts, no errors. Just predictions getting quietly worse.

How to detect it:

• Implement Population Stability Index (PSI) monitoring on input features
• Alert when PSI > 0.2 (significant drift)
• Compare weekly feature distributions against your training baseline

How to fix it:

• Set up automated retraining triggered by drift thresholds
• Maintain a rolling 6-12 month training window
• A/B test retrained models before deployment
• Document a data freshness SLA

2. Feedback Loop Amplification

HIGH

The recommendation engine only showed popular content, making it more popular, making the engine recommend it more. Over 6 months, recommendation diversity collapsed by 40%.

How to detect it:

• Track recommendation entropy, coverage, and novelty over time
• Monitor false negative rates across segments
• Measure exploration vs. exploitation ratio

How to fix it:

• Inject controlled randomization (epsilon-greedy or Thompson sampling)
• Maintain held-out evaluation sets NOT influenced by model output
• Implement counterfactual logging

3. Shadow Decision Boundaries

HIGH — Legal exposure

A credit scoring model effectively denied applicants from specific zip codes. Nobody programmed this — correlated features created a de facto discriminatory policy. The support team couldn't explain rejections because nobody knew.

How to detect it:

• Run SHAP/LIME on denial decisions monthly
• Segment outcomes by protected characteristics
• Create "model fact sheets" documenting effective decision rules

4. Cascade Failure Propagation

CRITICAL

A 5% accuracy drop in the upstream model caused 40% error downstream. Errors amplified through each pipeline stage. Root cause analysis was nearly impossible.

How to detect & fix:

• Map all model-to-model dependencies
• Circuit breakers between models
• Model SLAs with accuracy thresholds
• Graceful degradation to rules when ML is unreliable

5. Adversarial Vulnerability

CRITICAL

Small input perturbations flipped model decisions with >95% success rate. Fraudsters actively probe production systems to learn decision boundaries. They don't need model access — just feature understanding.

How to detect & fix:

• Run FGSM/PGD attacks on your own model
• Adversarial training in your pipeline
• Rate-limit API access for systematic probing
• Quarterly red team exercises