AI System Privacy Audit: Model Provider Risks and Mitigations¶

System in scope: doc_quality_compliance_check (backend + orchestrator + observability).

1. System Diagram¶

Model-relevant architecture facts used in this risk sheet:

• The backend/orchestrator uses a model adapter layer (Anthropic, OpenAI-compatible, Nemotron target/scaffold).
• Prompt/output traces are exposed and persisted for observability and audit workflows.
• The product already includes anti-hallucination signals and a validator stage, but model outputs can still be wrong or overconfident.

2. Data Flow Analysis¶

Corrected interpretation for privacy¶

• The critical privacy boundary is not only frontend-to-backend, but backend/orchestrator-to-model-provider.
• The highest-risk payload is the full prompt context plus model output, because both may contain direct or indirect personal data.
• Observability improves governance but increases privacy exposure if traces are over-retained or insufficiently redacted.

3. Sensitive Data¶

Sensitive Data: Prompt Context and Model Output Content¶

• Category: Input/output content containing direct or inferred personal data
• Examples: Names, emails, stakeholder assignments, reviewer identifiers, document passages copied into prompts, generated summaries that may restate personal data
• Why Sensitive: Leaves the primary application context during model inference (current external-provider path)
• Current Protection: Session auth, RBAC, TLS, audit events, optional graceful fallback without external model key
• Risk (or Harm) if Exposed: Unauthorized disclosure, regulatory non-compliance (GDPR purpose limitation/data minimization), reputational harm

Sensitive Data: Provider/Model Telemetry and Rich Trace Payload¶

• Category: Metadata linked to actors, workflows, and potentially sensitive content fingerprints
• Examples: provider, model_used, trace_id, correlation_id, latency/tokens, rich payload entries, prompt/output snapshots
• Why Sensitive: Enables reconstruction of who triggered what model action and when; can become re-identifiable when combined with audit tables
• Current Protection: Backend-protected endpoints, role checks, PostgreSQL persistence
• Risk (or Harm) if Exposed: Excessive internal visibility, inference attacks, unauthorized profiling of users/reviewers

Sensitive Data: Model Credentials and Routing Configuration¶

• Category: Secrets and control-plane configuration
• Examples: API keys, adapter routing flags, provider selection settings
• Why Sensitive: Compromise enables data exfiltration or unauthorized model usage
• Current Protection: Environment-based secret configuration and backend auth controls
• Risk (or Harm) if Exposed: Account abuse, data leakage, loss of integrity of compliance decisions

4. Privacy Risks¶

Risk 1: External model transfer of potentially personal prompt/output data¶

• Priority: High
• Risk Category: Model data transfer and residency
• Potential Harm/Impact: Data leaves controlled environment; third-party processing and cross-border concerns
• Ability to Implement Control: Medium
• Recommended controls:
• Replace external inference path with internal on-prem model gateway as default.
• Keep adapter interface stable so provider swap does not change business logic.
• Enforce strict egress policy: only approved internal model endpoints from production.

Risk 2: Over-collection and over-retention in model observability traces¶

• Priority: High
• Risk Category: Logging/telemetry minimization
• Potential Harm/Impact: Prompt/output traces may store unnecessary personal data for too long
• Ability to Implement Control: High
• Recommended controls:
• Apply pre-persistence redaction policy for prompt/output trace fields.
• Separate "operational" vs "audit-evidence" retention classes with explicit TTLs.
• Restrict raw trace visibility to least-privilege roles and log all access.

Risk 3: Hallucinated model output impacts compliance decisions¶

• Priority: High
• Risk Category: Model output reliability and decision integrity
• Potential Harm/Impact: Wrong legal/compliance guidance, missed obligations, false audit confidence
• Ability to Implement Control: Medium
• Recommended controls:
• Keep mandatory validator stage and add citation-required answer format.
• Add grounding checks against rule packs before output is accepted.
• Require HITL approval for high-impact outputs.

Risk 4: Model/version drift reduces reproducibility and explainability¶

• Priority: Medium
• Risk Category: Governance and traceability
• Potential Harm/Impact: Same input yields diverging output without clear reason; weak audit defensibility
• Ability to Implement Control: Medium
• Recommended controls:
• Persist model version, adapter version, prompt template version, and config hash for every run.
• Freeze approved model release sets per environment.
• Run periodic regression suite on reference compliance tasks.

Risk 5: Migration to on-prem models may degrade quality if unmanaged¶

• Priority: Medium
• Risk Category: Migration and model quality assurance
• Potential Harm/Impact: Quality drop or hidden failure modes could shift risk from privacy to governance failure
• Ability to Implement Control: High
• Recommended controls:
• Build a benchmark set from real workflows (privacy-scrubbed), including hallucination-sensitive cases.
• Use shadow mode (external vs on-prem) with acceptance thresholds before cutover.
• Define release gates: factuality, citation correctness, latency, and refusal behavior.

5. On-Prem Migration Plan (Quality-Preserving)¶

Target objective: remove external GenAI dependency for production personal-data paths while maintaining or improving compliance-output quality.

1. Adapter-first migration: keep orchestration and API contracts unchanged; switch routing in adapter layer.
2. Dual-run validation: run external and on-prem in parallel for a controlled period on the same scrubbed inputs.
3. Quality gate: promote on-prem only when benchmark thresholds are met (accuracy, hallucination rate, explainability/citations, latency).
4. Privacy hardening: enforce prompt minimization, redaction, and retention classes before and after cutover.
5. Operational fallback: if on-prem model confidence/validator checks fail, route to deterministic rules + HITL instead of external provider.

This sequence preserves output quality while reducing privacy exposure and external model dependency.

Mitigations and Controls¶

Appendix¶

Additional information from the repo source code

Subprocessor egress — Anthropic and Perplexity

Perplexity (src/doc_quality/services/research_service.py)¶

When: PERPLEXITY_API_KEY is non-empty.

Prompt content: Built from ResearchRequest: domain, description, target_market, and optional custom_query (or default research question) via prompts/research_prompt_v1.txt.

When key is empty: Static fallback; no HTTP call to Perplexity (still builds query string for ResearchResult).

Research API persistence (src/doc_quality/api/routes/research.py)¶

When result.provider == "perplexity", the API calls create_quality_observation with payload containing:

• llm_prompt (full built prompt),
• llm_output (full model answer),
• provider metadata and citation counts.

These are stored in quality_observations.payload (src/doc_quality/services/quality_service.py). This duplicates potentially sensitive user text on disk in addition to sending it to Perplexity.

Anthropic — risk template AI suggest (src/doc_quality/api/routes/risk_templates.py)¶

When: ANTHROPIC_API_KEY set; route ai-suggest.

Sent to Anthropic: System prompt (FMEA vs RMF), optional request.context (product/system context), partial_row as JSON, and instructions. Not full documents unless context or row fields contain them.

Anthropic — document analysis agent (src/doc_quality/agents/doc_check_agent.py)¶

Implementation: Sends up to 2000 characters of document content plus document_type and rule-based issues in the user message.

Wiring: DocumentCheckAgent is not referenced elsewhere in the Python tree (only defined in this file). The live analyze_document path uses document_analyzer.analyze_document only — rule-based, no LLM in current API flows. Treat as dead integration risk if something imports it later.

Compliance agent (src/doc_quality/agents/compliance_agent.py)¶

Creates an Anthropic client when a key exists but check_compliance delegates to compliance_checker only (no Anthropic messages observed in compliance_checker.py).

Orchestrator service (services/orchestrator/)¶

Separate process; AnthropicAdapter is scaffold-style. Boundary review: whatever workflows you enable may forward document fragments to backends — trace per workflow when operating this service.

Subprocessor summary¶