Automated compliance checks

Automated compliance checks at L4 go beyond the process gates of L3 (did the developer fill in the right fields?) to evaluate substantive compliance questions automatically: does this AI-generated code introduce patterns that are prohibited by the organization's security policy? Does this change affect a regulatory boundary in the system (a PCI scope crossing, a HIPAA data flow, an EU AI Act high-risk system component)? Does the model version used in this change have any known issues that require elevated review?

At L4 (Governed), automated compliance checks are running continuously - not just at PR time, but in the background as code is deployed, as regulatory guidance updates are published, and as new vulnerability disclosures affect AI-generated code patterns. The checks are not yes/no gates (that's L3) but scored assessments: this change has a compliance risk score of 73 based on its proximity to regulated data flows and the model version used. High-score changes get elevated review; low-score changes flow through automatically.

The technology stack for automated compliance checks at L4 typically includes: static analysis tools configured with compliance-specific rules (detecting data flow patterns that cross regulatory boundaries), AI-powered code review that applies policy rules at a semantic level (not just syntactic patterns), provenance analysis that correlates model version with compliance risk, and regulatory mapping tools that maintain a current map of which code modules are in regulatory scope.

One part of the agent stack has to be governed differently, and this edition states it plainly: skills and MCP servers are allowlisted and pinned, not scanned. On 2026-07-30 a single malicious skill was shown passing all eight of the available open-source skill scanners - Cisco's skill-scanner, NVIDIA SkillSpector, skillcheck, skillcop, claude-skill-antivirus and three others - using base64 and gzip wrappers, Cyrillic homoglyphs, paraphrasing and runtime command reassembly. Run in the other direction against 4,000 benign skills, six of those scanners flagged between 8.6% and 62.4% of them. A control with that error profile is not a control; it is a source of alerts. Allowlist the skills your organisation permits, pin them to specific versions, and re-review on upgrade.

There is now a framework for what that review should look at. The OWASP Agentic Skills Top 10 v1.0, published 2026-08-17, is the first security framework aimed at the skill layer specifically - SKILL.md, frontmatter, bundled scripts, the registry, inherited permissions - with categories AST01 to AST10 mapped across to AISVS, the NIST AI RMF and ISO 42001, which is what lets it slot into an existing compliance programme rather than sitting alongside one. It is grounded in a USENIX Security 2026 study of 98,380 marketplace skills that confirmed 157 malicious ones carrying 632 vulnerabilities, of which 73.2% used shadow features absent from their own documentation. That last number is the argument against documentation-based review as well as against scanning: what the skill says it does is not the population you are assessing.

MCP servers need the same treatment for the same reasons. The Splunk MCP Server app carried CVE-2026-76404 at CVSS 9.1, an unsafe deserialization leading to OS command execution, disclosed 2026-08-19 and fixed in 1.2.1 - pinning is what makes that patch a version bump rather than an archaeology exercise. And GhostSplice, published 2026-08-11, showed that fragmenting a single harmful instruction across tool descriptions and tool results raised compliance from 42% to 82% across eleven models, with Claude Haiku 4.5 holding at 0% through the API but reaching 100% inside Cursor. Test injection resistance in the IDE configuration you actually ship, not against the bare API.

The shift from L3 to L4 compliance checks is a shift in what's being evaluated. L3 checks are process checks: was the form filled in correctly? L4 checks are substantive checks: does the content of this change raise compliance concerns? L4 checks require understanding the semantics of code changes, not just their metadata. This is where AI-powered compliance review becomes self-referential: you're using AI to check whether AI-generated code meets compliance requirements.

• Process compliance is necessary but not sufficient - an AI-generated change can have complete MVAT fields, be reviewed by an approved reviewer, and still introduce a GDPR violation or a PCI scope crossing. Automated substantive checks catch what process checks cannot
• Regulatory boundaries are dynamic - as code evolves, what was outside regulatory scope can cross into scope without any developer deliberately intending it. Automated boundary tracking detects these crossings immediately rather than in annual security reviews
• AI-generated code may have systematic patterns - if a specific model version consistently generates a particular insecure pattern (e.g., a specific vulnerable serialization approach), automated scanning can detect all instances of that pattern across the codebase simultaneously
• Scales with AI-generated PR volume - as agents generate hundreds of PRs per day, human-dependent substantive compliance review cannot scale. Automated checks provide the substantive evaluation that allows auto-merge policies to operate safely
• Scanning the skill layer does not work yet - one file defeated all eight open-source skill scanners in July, and six of them flag between 8.6% and 62.4% of entirely benign skills; allowlisting and pinning are the controls that currently hold
• Documentation is not the artefact you are assessing - 73.2% of the confirmed-malicious skills in the USENIX corpus used shadow features that appear nowhere in their documentation, which is why review has to reach the bundled scripts and inherited permissions rather than the description
• Creates a risk-scored review queue - not all PRs need the same level of human review. Automated compliance scoring routes high-risk changes to expert reviewers and allows low-risk changes to merge automatically. This is how high-velocity AI-assisted delivery maintains compliance without review bottlenecks

1. Build a regulatory boundary map - before you can check whether changes cross regulatory boundaries, you need a map of where those boundaries are. This is an architecture exercise: for each regulatory regime that applies to your system (PCI, HIPAA, GDPR, SOC2), identify the specific modules, services, and data flows that are in scope. Store this map as a machine-readable configuration file (YAML or JSON) that compliance checks can read.
2. Implement data flow analysis - use static analysis tools (Semgrep with custom rules, CodeQL, or Datadog's code analysis) to detect data flows that cross regulatory boundaries. A GDPR check might detect when a new code path reads from the EU customer table and writes to an unencrypted log. A PCI check detects when PAN data flows to a new destination.
3. Configure AI-powered compliance review - integrate an AI reviewer (Claude, GPT-4) with your compliance ruleset as its system prompt. The AI reviewer evaluates PR diffs against the compliance rules and produces structured findings: {rule: "PCI-DSS 3.2.1", finding: "PANs may be logged in the new order processing path", severity: "high", location: "src/orders/processor.py:142"}. This is different from a general code review - it's specifically checking compliance rules.
4. Build the compliance risk score - aggregate individual check results into a per-PR score that routes the PR to the appropriate review tier: score 0-30 (auto-approvable), 31-70 (standard review), 71-100 (elevated review with compliance team involvement). The score weights different factors: regulatory boundary proximity, model version risk level, code area classification, and process compliance completeness.
5. Instrument for compliance dashboard - every automated compliance check run should produce events for the compliance dashboard: risk score distribution over time, which checks are triggering most frequently, which teams are generating the most elevated-review PRs. This dashboard drives the compliance improvement work queue.
6. Publish an allowlist of skills and MCP servers, pinned to versions - one list, owned by a named person, with a version pin per entry and a review on every upgrade. Enforce it in the client configuration through enterprise managed settings rather than by asking people to be careful, and refuse anything not on it. This is the control that replaces skill scanning, not a supplement to it.
7. Assess each entry against the OWASP Agentic Skills Top 10 - AST01 to AST10 give you the review checklist, and their mappings to AISVS, the NIST AI RMF and ISO 42001 let you file the results inside the compliance programme you already run. Review the bundled scripts and the inherited permissions, not just the SKILL.md and its frontmatter.
8. Test prompt-injection resistance in the client you actually ship - GhostSplice found the same model holding at 0% compliance through the API and reaching 100% inside an IDE. Your assessment result is a property of the configuration, not of the model, so run it against the IDE, the MCP servers and the skills your developers really have loaded.
9. Consider pre-install screening as a filter in front of the allowlist, not as the gate - published work on regex-plus-LLM screening reports F1 of 0.817 at a 1.13% false-positive rate using 77% fewer tokens than full-file screening, which is a useful triage step for deciding what is worth a human review. It is not a substitute for the allowlist.
10. Build the check update pipeline - compliance checks need to update as regulations change and as new AI vulnerability patterns are discovered. Build a pipeline where check updates go through a testing phase (run against the last 90 days of PRs to see what the new check would have caught) before deployment. This prevents compliance check updates from causing unexpected workflow disruptions.

Building checks that are too sensitive to false positives. A compliance check that flags every PR touching a payment-related file as "high risk" regardless of the actual change will be tuned out immediately. Compliance check precision matters more than recall: developers learn to ignore noisy checks, and noisy compliance checks undermine the credibility of the entire governance system.

Not maintaining the regulatory boundary map. The regulatory boundary map is only accurate at the moment it was created. As architecture evolves, new services are added, and data flows change, the map becomes stale. Stale maps produce false negatives: changes that cross regulatory boundaries are not detected because the boundary isn't mapped. Build a process for updating the boundary map when architecture changes, and include a "boundary map current?" check in the quarterly compliance review.

Treating AI compliance review as infallible. An AI-powered compliance reviewer is better than no automated check, but it makes mistakes. False negatives are particularly dangerous in compliance contexts. AI compliance review should be configured conservatively: when uncertain, flag for human review rather than passing. The AI reviewer is a triage tool, not a compliance certifier.

Not creating a feedback loop from production issues to checks. When a production security incident reveals a pattern that the automated checks missed, that pattern should become a new check. The compliance check suite should grow over time based on real findings, not just get built once and maintained unchanged. Build a post-incident process that includes "did any automated check detect this issue before it reached production? If not, can we add a check for this pattern?"

Treating a skill scanner's green result as an assurance. It is not one, and the evidence is unusually direct: a single crafted skill passed all eight open-source scanners tested in July. If a scanner is in your pipeline, treat its findings as leads for review and never as a clearance to install.

Allowlisting without pinning. An allowlist of names permits whatever the maintainer publishes next, which is exactly the failure mode behind every maintainer-account compromise of the last two months. Pin versions, and treat an upgrade as a new review rather than as a routine bump.

Assessing skills and servers in a lab configuration. Injection resistance changed completely between the API and an IDE in GhostSplice's testing - the same model, 0% and 100%. Whatever you conclude from testing a component in isolation says little about the assembled system your developers use.

Compliance checks that slow CI significantly. Complex static analysis and AI-powered review can add significant time to CI pipelines. Compliance checks that add more than 5 minutes to CI latency will generate pressure to disable or bypass them. Optimize for speed: run checks in parallel where possible, cache results for unchanged files, and scope checks to affected modules rather than running full-repo scans on every PR.