Basic audit: who uses what
Basic audit at L2 means the organization has established visibility into which developers are using which AI tools, at what frequency, and for what purposes.
- Official AI tool policy exists and is communicated to all developers
- The organization can list which developers use which AI tools
- The regulatory obligations that apply to this organisation's jurisdiction and sector are written down, with a named owner
- AI tool policy is reviewed at least annually
- Approved tool list is maintained and accessible
- Published AI tool policy document with distribution records
- AI tool usage tracking dashboard or report
What It Is
Basic audit at L2 means the organization has established visibility into which developers are using which AI tools, at what frequency, and for what purposes. It's not a comprehensive provenance record (that's L3-L4), but it answers the first-order governance question: who is using AI in our delivery pipeline, and are they using approved tools?
This level of audit typically draws on two sources: tooling vendor dashboards (GitHub Copilot usage stats, Claude for Teams analytics) and PR disclosure fields that developers fill in manually. Together, these give a reasonable picture of adoption without requiring custom instrumentation. The picture is incomplete - vendor dashboards show license seat usage and acceptance rates (adoption signals, not quality ones) but not the content of AI interactions; PR disclosures are self-reported and inconsistently filled - but it's a genuine improvement over the zero visibility of L1.
The audit data at L2 serves three purposes simultaneously. For compliance, it provides evidence of controlled AI use that can be presented in SOC2 or ISO 27001 audits. For management, it shows which teams have adopted AI tools and which haven't, enabling targeted support. For productivity analysis, it creates the first dataset correlating AI tool use with delivery metrics - even imperfect data starts to reveal patterns when collected consistently over time.
The critical discipline at L2 is to define audit questions before collecting data, not after. Organizations that collect everything they can access and then try to find meaning in it get drowned in noise. The useful audit questions at L2 are specific: Is every developer using an approved tool (not a personal subscription)? Are high-risk repositories showing appropriate AI disclosure rates? Is the distribution of AI use consistent with what developers report in surveys? Start with these questions and collect the data that answers them.
Why It Matters
- Makes compliance claims defensible - "we audit AI tool use and here is the data" is qualitatively different from "we have a policy" in an audit; actual usage data gives compliance claims evidentiary support
- Identifies shadow AI persistence - when vendor dashboard adoption numbers are lower than survey-reported AI use, the gap is shadow AI; the basic audit makes the gap visible and measurable rather than theoretical
- Reveals adoption patterns that inform support - some teams will adopt quickly, others will lag; the adoption map tells you where to focus enablement efforts and what barriers are preventing adoption in slower teams
- Creates the measurement baseline for ROI analysis - the same data that supports compliance audits also supports the business case for AI investment; correlating AI tool usage with PR throughput is the foundation of the productivity ROI argument
- Enables proactive issue detection - usage patterns that deviate from policy (tools being used that aren't on the approved list, usage patterns that suggest prohibited data types being processed) are visible in audit data before they become audit findings
Getting Started
- Consolidate vendor dashboards - every approved AI tool provides some usage analytics. GitHub Copilot has an organization-level dashboard showing seat usage, suggestion acceptance rates, and active users by team - all of them adoption signals, none of them evidence about the quality of what was accepted. Claude for Teams provides usage logs. Collect these into a single place - even a monthly spreadsheet - that gives an organization-wide view.
- Define your audit fields in PR templates - the PR audit fields you collect consistently are the most useful data. At minimum: AI tool used (dropdown from approved list, or "none"), purpose (code generation, test generation, code review, debugging, other), and rough percentage of code that was AI-generated. Keep it to three fields or developers won't fill it out.
- Create a compliance dashboard - a simple dashboard that shows: percentage of PRs with AI disclosure filled in, percentage of developers with active approved tool licenses, any tools appearing in disclosures that are not on the approved list. This dashboard is updated weekly and reviewed in your team leads meeting. Visibility creates accountability.
- Run a quarterly shadow AI re-census - vendor dashboard numbers tell you about approved tool use; they don't tell you about shadow AI. A quarterly anonymous survey ("are you using any AI tools not on the approved list?") tracks whether shadow AI is increasing, decreasing, or stable after the official policy was implemented.
- Flag anomalies for follow-up, not punishment - when audit data shows a developer using a non-approved tool, the first response should be a conversation: "we saw you used X, which isn't on our approved list - what were you trying to do, and does our approved list cover it?" This converts audit findings into policy improvement opportunities.
- Export audit data monthly for the compliance record - store monthly exports of usage data (team-level, not necessarily individual-level) in your compliance documentation repository. This creates the time series that auditors want to see: evidence of continuous monitoring, not just a snapshot at audit time.
The most useful metric in your basic audit is not adoption rate - it's the adoption gap between what developers report using and what you can see in official tooling. That gap is the shadow AI measurement and it should be shrinking quarter over quarter.
Common Pitfalls
Collecting audit data without acting on it. An audit dashboard that nobody looks at is theater, not governance. The audit data needs a review cadence - weekly for the compliance dashboard, monthly for the trend analysis - and someone accountable for following up on anomalies. Without action, the audit data creates false assurance.
Individual-level tracking that feels like surveillance. There's a meaningful difference between "12% of PRs from the backend team don't have AI disclosure fields filled in" (team-level, actionable) and "Sarah has only accepted 23% of Copilot suggestions this week" (individual-level, demotivating). Keep audit data at team level for management reporting. Individual-level data, if collected, should be accessible to the individual but not used in performance management.
Trusting vendor dashboards as complete. GitHub Copilot's dashboard shows suggestion acceptance rates, which count what a developer pressed Tab on rather than what survived - and it doesn't show chat usage, it doesn't show whether the developer used a personal account on the same task, and it doesn't show what code the suggestions were generated for. Vendor dashboards are one data source, not the complete picture. They should be supplemented with PR disclosure data.
Treating low adoption numbers as neutral. If 40% of licensed developers are actively using approved AI tools, that's not a neutral finding - it means 60% are either not using AI tools (lost productivity opportunity) or using shadow AI (governance risk). Low adoption numbers are always worth investigating, not just reporting.
Skipping the baseline. Organizations that start collecting audit data without establishing a baseline can't measure improvement. On the day you publish your AI policy, record: current adoption rate, shadow AI survey result, PR disclosure rate. That baseline is what you measure against at 90 days, 6 months, and 1 year.
How Different Roles See It
Bob has published the official AI tool policy and procured GitHub Copilot Enterprise for the team. Three months later, the CISO asks for evidence that the policy is being followed and that all AI use is through approved tools. Bob has the vendor dashboard but doesn't know how to answer the question "is anyone still using shadow AI?"
What Bob should do: Bob should pull three data sources together into a single report: Copilot Enterprise dashboard (active users by team, acceptance rates - read both as adoption evidence, which is what the CISO is asking about), PR template disclosure data from the last 90 days (what tools are being disclosed, what percentage of PRs have disclosures), and a fresh shadow AI survey (30-second anonymous survey sent via Slack: are you using any AI tools not provided by the company?). The triangulation of these three sources tells a much more credible story than any one of them alone. Bob should also note any discrepancies - if the shadow AI survey shows 15% still using personal tools, that's something to address, not hide. An honest report with an improvement plan is stronger than a polished report that doesn't acknowledge residual issues.
Sarah has access to the Copilot Enterprise dashboard and three months of PR disclosure data. She wants to use this to build the first AI productivity correlation report - but she's not sure her data is good enough to make meaningful claims.
What Sarah should do: Sarah should start with the most conservative claim the data supports. She can show: teams with more weekly active AI users have X% higher PR throughput in the same period. This correlation is imperfect (many confounding variables) but directionally meaningful. She should leave acceptance rate out of the correlation entirely - it counts accepted suggestions, roughly a third of which are deleted outright and mostly within fifteen minutes, so it cannot carry a claim about output. Sarah should present it as a correlation, not a causal claim, with the caveat that it's a starting point for a more rigorous analysis. More importantly, she should use the presentation to make the case for better data: "to answer this question more definitively, we need X months of structured data from the audit fields we added to the PR template." The imperfect first analysis builds the case for the more robust longitudinal analysis.
Victor is the heaviest AI user on the team - he is active in the tooling every working day, his PR throughput is high, and his code quality metrics are strong. But he's also done things the audit doesn't capture: running Claude Code sessions that produce entire modules, using AI for architecture review, using agent workflows that touch multiple repositories. The basic audit doesn't have a good category for his actual workflow.
What Victor should do: Victor should work with Sarah to improve the PR disclosure fields to capture the patterns the current schema misses. The current "code generation / test generation / code review / other" taxonomy doesn't distinguish between copilot-style suggestions and agent-generated PRs - a distinction that matters for both productivity analysis and compliance. Victor should propose an updated taxonomy that captures: suggestion-level assistance, chat-level assistance, and agent-generated code. He can validate the taxonomy against his own workflow first, then bring it to the team as a proposal. Better taxonomy makes the audit data more useful for everyone.
Further Reading
How This Guide Changed
What each edition changed in this guide, newest first.
- V1.6September 2026LATEST
Vendor dashboards stayed in the guide, but their acceptance-rate figures are now labelled for what they are: evidence that people are using the tools, not evidence that what they accepted was any good. The audit this level asks for is an adoption audit, and reading a Copilot acceptance rate as a quality result is the quickest way to hand a CISO a number that does not mean what the room thinks it means.
- V1.3June 2026
Vanta reorganised its evidence-collection material and the citation moved with it; the guide's own advice was untouched. June's governance news was about pricing restructuring turning into a procurement risk, a different question from this one, and the L2 audit picture stayed where it stood.
- V1.0March 2026
Written for the first edition as the modest first step in AI governance - not provenance, not attribution, just an honest answer to who is using which tool and how often. Its main contribution was a discipline rather than a tool: decide the audit questions before collecting anything, because teams that start from the vendor dashboards collect everything and learn nothing.
Where does your team actually sit on this?
This guide describes one level of one area. Run the assessment to place your team across all 16 areas, see which gates you have passed, and get a report you can take to your stakeholders.
Governance & Compliance