deliveryL2 GuidedGovernance & Compliance

Basic audit: who uses what

Basic audit at L2 means the organization has established visibility into which developers are using which AI tools, at what frequency, and for what purposes.

·Official AI tool policy exists and is communicated to all developers
·Basic audit tracking is in place (which developers use which AI tools)
·EU AI Act awareness training or briefing has been conducted

·AI tool policy is reviewed at least annually
·Approved tool list is maintained and accessible

Evidence

·Published AI tool policy document with distribution records
·AI tool usage tracking dashboard or report
·EU AI Act training completion records

What It Is

Basic audit at L2 means the organization has established visibility into which developers are using which AI tools, at what frequency, and for what purposes. It's not a comprehensive provenance record (that's L3-L4), but it answers the first-order governance question: who is using AI in our delivery pipeline, and are they using approved tools?

This level of audit typically draws on two sources: tooling vendor dashboards (GitHub Copilot usage stats, Claude for Teams analytics) and PR disclosure fields that developers fill in manually. Together, these give a reasonable picture of adoption without requiring custom instrumentation. The picture is incomplete - vendor dashboards show license seat usage and acceptance rates but not the content of AI interactions; PR disclosures are self-reported and inconsistently filled - but it's a genuine improvement over the zero visibility of L1.

The audit data at L2 serves three purposes simultaneously. For compliance, it provides evidence of controlled AI use that can be presented in SOC2 or ISO 27001 audits. For management, it shows which teams have adopted AI tools and which haven't, enabling targeted support. For productivity analysis, it creates the first dataset correlating AI tool use with delivery metrics - even imperfect data starts to reveal patterns when collected consistently over time.

The critical discipline at L2 is to define audit questions before collecting data, not after. Organizations that collect everything they can access and then try to find meaning in it get drowned in noise. The useful audit questions at L2 are specific: Is every developer using an approved tool (not a personal subscription)? Are high-risk repositories showing appropriate AI disclosure rates? Is the distribution of AI use consistent with what developers report in surveys? Start with these questions and collect the data that answers them.

Why It Matters

Makes compliance claims defensible - "we audit AI tool use and here is the data" is qualitatively different from "we have a policy" in an audit; actual usage data gives compliance claims evidentiary support
Identifies shadow AI persistence - when vendor dashboard adoption numbers are lower than survey-reported AI use, the gap is shadow AI; the basic audit makes the gap visible and measurable rather than theoretical
Reveals adoption patterns that inform support - some teams will adopt quickly, others will lag; the adoption map tells you where to focus enablement efforts and what barriers are preventing adoption in slower teams
Creates the measurement baseline for ROI analysis - the same data that supports compliance audits also supports the business case for AI investment; correlating AI tool usage with PR throughput is the foundation of the productivity ROI argument
Enables proactive issue detection - usage patterns that deviate from policy (tools being used that aren't on the approved list, usage patterns that suggest prohibited data types being processed) are visible in audit data before they become audit findings

Getting Started

Consolidate vendor dashboards - every approved AI tool provides some usage analytics. GitHub Copilot has an organization-level dashboard showing seat usage, suggestion acceptance rates, and active users by team. Claude for Teams provides usage logs. Collect these into a single place - even a monthly spreadsheet - that gives an organization-wide view.
Define your audit fields in PR templates - the PR audit fields you collect consistently are the most useful data. At minimum: AI tool used (dropdown from approved list, or "none"), purpose (code generation, test generation, code review, debugging, other), and rough percentage of code that was AI-generated. Keep it to three fields or developers won't fill it out.
Create a compliance dashboard - a simple dashboard that shows: percentage of PRs with AI disclosure filled in, percentage of developers with active approved tool licenses, any tools appearing in disclosures that are not on the approved list. This dashboard is updated weekly and reviewed in your team leads meeting. Visibility creates accountability.
Run a quarterly shadow AI re-census - vendor dashboard numbers tell you about approved tool use; they don't tell you about shadow AI. A quarterly anonymous survey ("are you using any AI tools not on the approved list?") tracks whether shadow AI is increasing, decreasing, or stable after the official policy was implemented.
Flag anomalies for follow-up, not punishment - when audit data shows a developer using a non-approved tool, the first response should be a conversation: "we saw you used X, which isn't on our approved list - what were you trying to do, and does our approved list cover it?" This converts audit findings into policy improvement opportunities.
Export audit data monthly for the compliance record - store monthly exports of usage data (team-level, not necessarily individual-level) in your compliance documentation repository. This creates the time series that auditors want to see: evidence of continuous monitoring, not just a snapshot at audit time.

Tip

The most useful metric in your basic audit is not adoption rate - it's the adoption gap between what developers report using and what you can see in official tooling. That gap is the shadow AI measurement and it should be shrinking quarter over quarter.

6 steps to get from here to the next level

Common Pitfalls

Collecting audit data without acting on it. An audit dashboard that nobody looks at is theater, not governance. The audit data needs a review cadence - weekly for the compliance dashboard, monthly for the trend analysis - and someone accountable for following up on anomalies. Without action, the audit data creates false assurance.

Individual-level tracking that feels like surveillance. There's a meaningful difference between "12% of PRs from the backend team don't have AI disclosure fields filled in" (team-level, actionable) and "Sarah has only accepted 23% of Copilot suggestions this week" (individual-level, demotivating). Keep audit data at team level for management reporting. Individual-level data, if collected, should be accessible to the individual but not used in performance management.

Trusting vendor dashboards as complete. GitHub Copilot's dashboard shows suggestion acceptance rates - it doesn't show chat usage, it doesn't show whether the developer used a personal account on the same task, and it doesn't show what code the suggestions were generated for. Vendor dashboards are one data source, not the complete picture. They should be supplemented with PR disclosure data.

Treating low adoption numbers as neutral. If 40% of licensed developers are actively using approved AI tools, that's not a neutral finding - it means 60% are either not using AI tools (lost productivity opportunity) or using shadow AI (governance risk). Low adoption numbers are always worth investigating, not just reporting.

Skipping the baseline. Organizations that start collecting audit data without establishing a baseline can't measure improvement. On the day you publish your AI policy, record: current adoption rate, shadow AI survey result, PR disclosure rate. That baseline is what you measure against at 90 days, 6 months, and 1 year.

Mistakes teams actually make at this stage - and how to avoid them

How Different Roles See It

BobHead of Engineering

Bob has published the official AI tool policy and procured GitHub Copilot Enterprise for the team. Three months later, the CISO asks for evidence that the policy is being followed and that all AI use is through approved tools. Bob has the vendor dashboard but doesn't know how to answer the question "is anyone still using shadow AI?"

What Bob should do: Bob should pull three data sources together into a single report: Copilot Enterprise dashboard (active users by team, acceptance rates), PR template disclosure data from the last 90 days (what tools are being disclosed, what percentage of PRs have disclosures), and a fresh shadow AI survey (30-second anonymous survey sent via Slack: are you using any AI tools not provided by the company?). The triangulation of these three sources tells a much more credible story than any one of them alone. Bob should also note any discrepancies - if the shadow AI survey shows 15% still using personal tools, that's something to address, not hide. An honest report with an improvement plan is stronger than a polished report that doesn't acknowledge residual issues.

What Bob should do - role-specific action plan

SarahProductivity Lead

Sarah has access to the Copilot Enterprise dashboard and three months of PR disclosure data. She wants to use this to build the first AI productivity correlation report - but she's not sure her data is good enough to make meaningful claims.

What Sarah should do: Sarah should start with the most conservative claim the data supports. She can show: teams with higher Copilot acceptance rates have X% higher PR throughput in the same period. This correlation is imperfect (many confounding variables) but directionally meaningful. Sarah should present it as a correlation, not a causal claim, with the caveat that it's a starting point for a more rigorous analysis. More importantly, she should use the presentation to make the case for better data: "to answer this question more definitively, we need X months of structured data from the audit fields we added to the PR template." The imperfect first analysis builds the case for the more robust longitudinal analysis.

What Sarah should do - role-specific action plan

VictorStaff Engineer - AI Champion

Victor is the heaviest AI user on the team - his acceptance rate is high, his PR throughput is high, and his code quality metrics are strong. But he's also done things the audit doesn't capture: running Claude Code sessions that produce entire modules, using AI for architecture review, using agent workflows that touch multiple repositories. The basic audit doesn't have a good category for his actual workflow.

What Victor should do: Victor should work with Sarah to improve the PR disclosure fields to capture the patterns the current schema misses. The current "code generation / test generation / code review / other" taxonomy doesn't distinguish between copilot-style suggestions and agent-generated PRs - a distinction that matters for both productivity analysis and compliance. Victor should propose an updated taxonomy that captures: suggestion-level assistance, chat-level assistance, and agent-generated code. He can validate the taxonomy against his own workflow first, then bring it to the team as a proposal. Better taxonomy makes the audit data more useful for everyone.

What Victor should do - role-specific action plan