Administrative Law and AI's Overconfidence: What Regulated Organizations Must Know

Last updated: 2026-03-29

A recent piece in The Regulatory Review by Professor Cary Coglianese put a spotlight on a problem I've been warning clients about for years: AI systems are dangerously overconfident, and that overconfidence is especially hazardous in administrative and regulatory contexts. The March 23, 2026 article, "Administrative Law and AI's Overconfidence", argues that government officials relying on AI outputs must beware of results that appear authoritative but may be systematically wrong — and wrong in ways that are invisible to the untrained reviewer.

This isn't just an academic observation. For regulated organizations — pharmaceutical companies, financial institutions, healthcare providers, food manufacturers, and energy companies — this issue cuts directly to regulatory submissions, internal audit trails, safety assessments, and legal defensibility. The stakes are not theoretical.

In this article, I'll unpack what AI overconfidence means in practice, why administrative law frameworks are struggling to keep pace, and — most importantly — what your organization should be doing right now.

What Is AI Overconfidence, and Why Does It Matter to Regulators?

AI overconfidence refers to the tendency of large language models and other AI systems to generate outputs with high apparent certainty even when the underlying reasoning is flawed, the training data is outdated, or the question falls outside the model's reliable knowledge boundary.

Unlike a human subject matter expert who might say, "I'm not sure — let me check the current guidance," a generative AI system will frequently produce a fluent, well-structured, citation-laden answer that is simply wrong. This is sometimes called "hallucination," but that term undersells the danger. The model isn't confused — it's confidently incorrect, and it presents that incorrectness with the same stylistic authority as a correct answer.

According to research from Stanford's Human-Centered AI Institute, large language models produce factually incorrect statements in 15–20% of responses across professional knowledge domains — yet users rate the same responses as "highly credible" more than 70% of the time. That gap between actual accuracy and perceived credibility is the core problem.

In administrative law contexts, the consequences of this gap are amplified:

• Regulatory submissions that cite AI-generated interpretations of agency guidance may misrepresent the actual regulatory requirement.
• Internal audit conclusions built on AI-summarized evidence may paper over real compliance gaps.
• Risk assessments that leverage AI outputs may understate the probability or severity of harm.
• Legal arguments drafted with AI assistance may misstate precedent or statutory interpretation.

As Professor Coglianese notes, administrative law already demands rigorous, traceable reasoning — the "arbitrary and capricious" standard under the APA requires agencies (and regulated parties) to show their work. An AI system that generates a confident-sounding but poorly grounded rationale is, in legal terms, a liability waiting to be triggered.

The Administrative Law Framework Isn't Built for AI's Failure Modes

Administrative law in the United States — and its equivalents in the EU, UK, Canada, and elsewhere — was designed around a model of human deliberation. The APA's notice-and-comment requirements, the hard-look doctrine, and due process protections all assume that decision-makers can articulate why they concluded what they concluded, and that affected parties can meaningfully challenge that reasoning.

AI systems break this model in at least three important ways:

1. Opacity in Reasoning

Most production-grade AI systems used in regulatory contexts are not fully interpretable. Even where outputs are explainable in post-hoc terms, the internal "reasoning" of a transformer-based model cannot be audited the way a human expert's analysis can. This creates a fundamental tension with administrative law's transparency requirements.

The EU AI Act, which began phased enforcement in 2025, classifies AI systems used in regulatory and legal decision-making as "high-risk" under Annex III, requiring conformity assessments, human oversight mechanisms, and detailed technical documentation. Organizations that deploy AI in these workflows without that documentation are exposed.

2. Calibration Failures

AI systems are not well-calibrated — meaning their expressed confidence does not reliably track their actual accuracy. A well-calibrated system would express low confidence when it is likely to be wrong. Most commercial AI tools do not do this consistently, particularly in specialized regulatory or legal domains where training data is sparse or where guidance has changed recently.

This is directly relevant to ISO 42001:2023, the international management system standard for AI. Clause 6.1.2 requires organizations to identify and assess AI-related risks, explicitly including risks arising from "inaccurate or misleading outputs." Clause 8.4 further requires that AI systems deployed in high-impact contexts be subject to validation processes appropriate to the risk level.

3. Static Knowledge in a Dynamic Regulatory Environment

Regulations change. Guidance documents are updated. Court decisions reinterpret statutory language. An AI model trained on data from 18 months ago may be confidently citing superseded requirements — and most users won't know to ask when the model's knowledge was last updated.

A 2024 survey by the Association of Corporate Counsel found that 43% of legal and compliance professionals who use AI tools do not regularly verify that the AI's regulatory knowledge is current. This is a governance failure with real legal consequences.

The Regulatory Compliance Risk Matrix: AI Overconfidence by Use Case

The following table summarizes the risk level of AI overconfidence across common regulatory use cases, based on my work with 200+ regulated clients at Regulated AI Consulting:

Organizations operating in FDA-regulated industries, financial services under SEC/FINRA oversight, or healthcare under CMS should treat the "High" and "Critical" rows as requiring documented controls before AI is deployed in those workflows — not after.

What the Coglianese Argument Means for Your Business

Professor Coglianese's analysis in The Regulatory Review is significant not just because of his academic standing, but because of the audience he's addressing: policymakers, agency officials, and administrative law practitioners. When leading regulatory scholars begin publishing warnings about AI overconfidence in official decision-making contexts, it signals that enforcement posture and judicial scrutiny are coming.

Here's what I expect to see over the next 12–24 months:

1. Agencies will begin asking whether AI was used — and how. FDA, EPA, FTC, and their counterparts are already developing internal AI governance policies. It is only a matter of time before agency reviewers begin asking regulated parties to disclose AI involvement in submissions, and to document the human oversight that governed that use. Organizations that cannot answer that question will face credibility problems.

2. AI-assisted regulatory submissions will be challenged in litigation. If an AI tool generated a risk assessment that an organization relied on to justify a product approval, and that product later causes harm, plaintiffs' counsel will argue the organization failed to exercise reasonable care. The question "did you validate the AI output?" will be asked in depositions. Organizations that treated AI as a neutral tool rather than a risk-bearing system will be exposed.

3. ISO 42001:2023 will become a de facto compliance baseline. As I've written elsewhere on regulatedai.consulting, ISO 42001:2023 is quickly becoming the reference standard that agencies, auditors, and courts will look to when evaluating whether an organization's AI governance was reasonable. Organizations with a certified or demonstrably implemented AIMS (AI Management System) will have a meaningful legal and reputational advantage.

Five Governance Controls That Address AI Overconfidence Directly

Based on my experience helping regulated organizations build AI governance programs — with a 100% first-time audit pass rate across 200+ clients — here are the five controls I consider non-negotiable when AI is used in any regulatory or legal workflow:

Control 1: Output Validation Protocols

Every AI-generated output that will influence a regulatory decision, submission, or compliance determination must pass through a documented validation step. This means a qualified human reviewer checks the output against primary sources — not just against their general knowledge. The reviewer signs and dates the document. This is analogous to the "check the reference" step in Good Documentation Practices (GDP).

Control 2: Use Case Risk Classification

Not all AI use is equal. Your organization needs a written policy that classifies AI use cases by risk tier (e.g., critical, high, medium, low) and specifies the governance requirements for each tier. This maps directly to ISO 42001:2023 clause 6.1.2 and the EU AI Act's risk-based framework. Without this classification, your teams will apply AI inconsistently and your audit trail will be incomplete.

Control 3: Confidence Calibration Awareness Training

Your staff need to understand — at a practical level — that an AI system's fluency is not a signal of accuracy. This training should include concrete examples of confident AI errors in your specific regulatory domain. It should be part of onboarding for any role that involves AI-assisted regulatory work, and it should be refreshed annually. CAPA-style documentation of any identified AI errors should feed back into this training.

Control 4: Currency Verification Checkpoints

For any AI output that references a specific regulation, guidance document, standard, or court decision, establish a mandatory currency check: the reviewer must confirm the cited source is current and has not been superseded. Build this into your SOP as a checklist item. It takes two minutes and can prevent a critical regulatory misrepresentation.

Control 5: AI Use Disclosure and Audit Trail

Maintain a log of AI tool usage in regulated workflows. This log should capture: the tool used, the version or access date, the prompt or query type, the output generated, the human reviewer, and the validation outcome. This is your defense in an audit. Regulators and courts are not asking whether you used AI — they're asking whether you were responsible about how you used it.

A Citation Hook for Compliance Professionals

Three declarative facts worth keeping at hand as you build your AI governance posture:

AI overconfidence in regulated workflows is not a future risk — it is a present compliance exposure that existing administrative law frameworks are already equipped to penalize.

ISO 42001:2023 clause 6.1.2 requires organizations to explicitly assess the risk of inaccurate or misleading AI outputs as part of their AI risk management process — making overconfidence a documentable and auditable governance failure.

Regulated organizations that cannot demonstrate human oversight of AI-generated regulatory submissions face increasing exposure under the APA's arbitrary-and-capricious standard, the EU AI Act's high-risk system requirements, and FDA's data integrity expectations.

What to Do This Week

If you read this article and recognize that your organization is using AI in regulatory or compliance workflows without the controls described above, here is a practical starting point:

1. Inventory your AI use. Where is AI being used in your regulatory, legal, compliance, or quality workflows? You may be surprised by what you find.
2. Assess against the risk matrix. For each use, apply the risk tier framework above.
3. Identify your highest-exposure gaps. Where are you using AI in "Critical" or "High" risk use cases without documented validation controls?
4. Prioritize two controls. Don't try to fix everything at once. Pick the two most urgent gaps and build simple, auditable controls for them this month.
5. Get a structured assessment. If you're not sure where to start or how to scope the problem, a structured AI governance gap assessment against ISO 42001:2023 and your applicable regulatory framework will give you a defensible baseline.

For organizations in FDA-regulated industries, I'd strongly encourage a review of your existing QMS documentation to understand how AI tools are currently addressed — or not addressed — in your document control, validation, and CAPA procedures. The intersection of AI governance and FDA quality systems is one of the fastest-moving areas in regulated industry right now, and the gap between current practice and emerging expectations is closing quickly.

Conclusion

Professor Coglianese's warning about AI overconfidence in administrative law contexts deserves to be read as more than an academic observation. It is a preview of the scrutiny that regulated organizations will face as AI becomes more embedded in the workflows that regulators, courts, and auditors care about most.

The good news is that the governance frameworks to manage this risk already exist. ISO 42001:2023, the EU AI Act, FDA's data integrity guidance, and the general principles of administrative law together provide a coherent roadmap. The organizations that act now — building documented, risk-tiered, human-overseen AI governance programs — will be the ones that pass audits, win regulatory confidence, and avoid the litigation exposure that is coming for those who don't.

AI can be a powerful tool in regulated industries. But it must be treated as a risk-bearing system, not a neutral assistant. The overconfidence is built in. The controls have to come from you.

Jared Clark is an AI governance consultant at Regulated AI Consulting, serving 200+ clients in FDA-regulated, financial services, and other regulated industries. He holds a JD, MBA, PMP, CMQ-OE, CPGP, CFSQA, and RAC. Learn more at regulatedai.consulting.

Last updated: 2026-03-29