How Artificial Intelligence Is Reshaping Federal Agency Power, and Why the Law Has Not Caught Up
By Malcolm Lee Kitchen III | Margin Of The Law
Federal agencies are building systems that decide who gets disability benefits, who gets flagged for securities fraud, and who gets investigated by the government. They are doing this with artificial intelligence. Not with human judgment run through a computer. With machine learning models that process data, generate predictions, and increasingly drive outcomes that affect real people.
This is not a thought experiment. It is happening now. And the legal framework designed to hold agencies accountable has not kept pace.
This article documents what is actually deployed at two federal agencies, the Social Security Administration and the Securities and Exchange Commission, examines why existing administrative law is poorly equipped to manage it, and proposes a concrete accountability mechanism that can work within the constraints of how government actually operates.
The Setup: Government Has Been Here Before
In 1993, President Clinton stood in Sunnyvale, California, and promised to reinvent government through data and performance measurement. Sunnyvale had built a system of quantified targets, performance-based pay, and continuous self-evaluation. Clinton and Vice President Gore held it up as the model for a leaner, more responsive public sector.
It collapsed within a decade. When the performance index dropped, officials changed the weights. When new weights did not fix anything, they scrapped the measure. By 1999, employees were leaving in large numbers and accusing leadership of mismanagement. What started as a framework for accountability became a system that agencies could game, adjust, and ultimately hollow out.
IBM published a white paper in 2018 making nearly identical promises. Artificial intelligence, the paper claimed, would reinvent government, make it more client-oriented, improve decision making for civil servants, and deliver greater transparency. The pitch is familiar. The overconfidence is familiar. The gap between the promise and the operational reality is also familiar.
The difference this time is the stakes. The algorithmic governance tools now entering federal agencies are not internal performance trackers. They are being applied to the redistributive and coercive functions of the state. They determine who gets benefits and who gets investigated. When those tools fail, the consequences are not a bad metric. They are a denied disability claim, a wrongly targeted investment advisor, a feedback loop that concentrates enforcement on the same subset of actors while others evade detection.
To understand what accountability requires, you need to understand what is actually being deployed.
What the Social Security Administration Is Actually Running
The SSA adjudicates disability benefits through a system of administrative law judges. The problem those judges have created is well documented. Cases are randomly assigned. Outcomes are not random.
Figure the data: in 2018, one judge in a single region awarded benefits in 8 percent of cases. Another judge in the same region awarded benefits in 98 percent of cases. Cases were randomly assigned within the office. The variation cannot be explained by case composition. It reflects the discretion, biases, and inconsistencies of individual judges. People with identical medical conditions are getting opposite outcomes based on which judge happens to receive their file.
Jerry Mashaw documented this problem forty years ago. It persists today. Backlogs at the SSA, the Board of Veterans Appeals, the Office of Medicare Hearings and Appeals, and the Executive Office for Immigration Review have reached levels that effectively deny timely access to benefits that Congress authorized.
The SSA Appeals Council has developed three machine learning applications to address these failures.
The first is a clustering algorithm. Because cases were randomly assigned, judges were constantly switching between different areas of disability law. Each area has its own decision trees, with roughly 2,000 possible paths through a typical disability case. The clustering model uses information about claimant age, functional impairments, and state of origin to group similar cases together. Judges working on cases in the same cluster develop familiarity with the same portion of the decision tree. The system created micro-specialization within each judge’s docket without changing how cases were assigned across judges. Early pilots reported a 7 percent gain in productivity and a 12.5 percent reduction in errors.
The second is a Quick Disability Determination model. The SSA finalized a rule in 2010 enabling expedited processing for cases likely to result in a full grant of benefits. The model draws on medical history, treatment protocols, symptoms, and clinical findings to predict which cases are clear grants. Those cases go to a QDD review team rather than through a full hearing. The model uses Naive Bayes classification with state-level information to identify fully favorable dispositions. The QDD team reviews the prediction and can recommend an expedited grant.
The third, and most technically sophisticated, is the Insight system. Developed internally by an attorney-turned-analyst at SSA, Insight draws on decades of decision trees and policy documentation. It uses natural language processing, including regular expressions, semantic parsing, and supervised classification, to flag potential errors in draft decisions before they are finalized. As one concrete example: Insight extracts a claimant’s listed functional impairments and checks whether the impairment is consistent with the job classifications in the Department of Labor’s Dictionary of Occupational Titles. If an ALJ’s draft decision assigns a claimant to a job category that does not match their documented limitations, Insight flags the inconsistency. The system was adopted voluntarily at the Appeals Council in 2016 and at hearing offices in 2017. Early results showed reductions in processing time and in decisions returned to judges for error correction.
These tools represent real progress on a decades-old problem. Faster, more consistent processing could reduce backlogs that force claimants to wait years for hearings. Better error detection could catch mistakes before they generate costly appeals. If the tools improve accuracy, they might also reclaim part of what due process was supposed to provide: a meaningful assessment of whether a claimant qualifies for benefits.
But the risks are real.
Automation bias is the documented tendency of human decision makers to defer excessively to automated outputs. An ALJ presented with a predicted disposition score may compare their own judgment against the machine’s recommendation and, over time, give that recommendation more weight than it deserves. The search tool may allow a judge to spend less time reviewing the full record. A machine-generated decision template may convert the ALJ’s role from drafting to signing. The formal requirement of human review does not guarantee that review is meaningful.
There is also a centralization risk. If tools like Insight allow agency leadership to encode specific policy choices into how cases are flagged, adjudication that was intended to be independent can become an extension of executive policy. In immigration adjudication, the exemption of immigration judges from performance reviews was secured by letter, not statute. It was later removed, enabling greater presidential control over adjudicatory outcomes. Algorithmic tools can accelerate this kind of control without requiring any formal rule change.
And the evidence base for these tools is weaker than the SSA’s internal reports suggest. Usage of Insight was voluntary. Employees who adopted the system may simply have been more motivated or more careful than those who did not. SSA’s own Inspector General acknowledged this problem, noting that management had not defined objectives in measurable terms that would allow actual performance assessment. The gains attributed to these tools may be real. They may also reflect selection effects that have nothing to do with the algorithms themselves.
What the Securities and Exchange Commission Is Actually Running
Agency enforcement poses a structural problem. Resources are finite. Regulatory targets are not. The SEC is responsible for monitoring tens of thousands of investment advisors, broker-dealers, mutual funds, and exchange-traded funds, plus the full range of trading activity across U.S. equity and options markets. No manual process can cover that ground at scale.
The SEC has built a suite of algorithmic tools to address this.
The first two focus on trading-based misconduct. ARTEMIS, the Advanced Relational Trading Enforcement Metrics Investigation System, analyzes patterns across more than six billion electronic equities and options trading records. It is designed to identify serial offenders, traders with histories of suspicious activity around significant corporate events. The system begins by pulling 8-K filings, the SEC’s required disclosures for material events, and running them through two separate tools. The first is a natural language processing topic model that classifies filings into categories: mergers and acquisitions activity, CEO terminations, FDA approval decisions, bankruptcy announcements, and others. The second is a supervised learning algorithm that flags current filings and trigger events for further review.
Once a filing is flagged, an analyst reviews the results. If further investigation seems warranted, the analyst submits a bluesheet request, a statutory investigatory tool that compels detailed trading data from broker-dealers. That data includes standard transaction information plus personal information about trading participants. ARTEMIS then analyzes the bluesheet data against records from prior requests to identify suspicious anomalies.
ATLAS, the Abnormal Trading and Link Analysis System, complements ARTEMIS by focusing on first-time insider trading rather than serial offenders. It uses a supervised model called a one-class support vector machine. Potential targets are divided into two categories: those who lost money on a trade and those who made money. The SVM is trained on those who lost money, then applied to those who made money. The underlying assumption is that profitable traders should not behave fundamentally differently from unprofitable ones over time. Outliers flagged by both tools are treated as candidates for investigation.
The third tool targets investment advisor compliance. The SEC requires advisors to file Form ADV disclosures, which include free-text narratives about their business practices, disciplinary history, conflicts of interest, and client base. The SEC’s NLP-based tool processes these disclosures in three steps: extracting text from PDF forms and segmenting it into sections that correspond to specific questions, running unsupervised learning to cluster document types and detect anomalies, and then applying supervised learning using prior Form ADVs associated with enforcement referrals to classify each investment advisor as high, medium, or low risk. High-risk advisors are passed to enforcement staff along with an explanation identifying which features drove the classification.
These tools represent a genuine advance in regulatory capacity. The SEC can now sift through volumes of data that no team of human investigators could process manually. The tools can identify suspicious patterns that would never surface through conventional investigation. They can focus scarce enforcement resources on the highest-risk targets. They can narrow the technology gap between an underfunded regulatory agency and well-resourced financial institutions with armies of compliance professionals.
But the problems are structural and significant.
The bluesheet data that feeds ARTEMIS and ATLAS is not a random sample. It reflects prior enforcement decisions, which reflect the assumptions, heuristics, and biases of enforcement staff. When the models train on biased data, they learn to replicate and amplify those biases. The IRS has historically addressed this problem through random audit data that provides a genuine baseline. No such baseline exists at the SEC.
The enforcement feedback loop creates additional risk. When line-level staff retain authority to initiate investigations, and algorithmic tools flag cases based on patterns from prior enforcement actions, the tools can become self-reinforcing. If the model learned to flag a specific network of investment advisors because that network was heavily investigated last year, it will continue flagging similar networks regardless of whether underlying behavior has changed. Human reviewers update based on new information. A model trained on past referrals may not.
Gaming is a concrete and underappreciated threat. Sophisticated regulated parties can reverse-engineer enforcement models. Major financial institutions have quantitative staff who can test how their disclosures might be classified, identify the features that drive risk scores, and adjust their language accordingly. The SEC’s NLP model can be fooled by adding text that does not change meaning for a human reader but shifts a classification from high risk to low. That technique is well documented in computer science research. It is well within the capacity of sophisticated actors to deploy.
The dynamic nature of financial wrongdoing creates a final challenge. Enforcement tools are trained on past patterns. Wrongdoing evolves. New instruments, new structures, and new evasion techniques emerge continuously. A model that can flag the transaction pattern used to implement last year’s illegal tax shelter will not necessarily recognize a new variant designed to stay just outside the model’s detection boundary. Keeping tools current requires continuous investment in model development that most agencies cannot sustain.
Why Current Administrative Law Cannot Handle This
Federal administrative law is built on two pillars: transparency and reason giving. When government takes action that affects rights, it must explain why. The Administrative Procedure Act encodes this requirement throughout. Agency decisions must include findings and conclusions and the reasons behind them. Courts can strike down agency action that is arbitrary or capricious.
Advanced machine learning is structurally resistant to this requirement. The most sophisticated models cannot be fully explained even by their own engineers. A deep learning model may produce accurate predictions without any interpretable logic behind them. The rules a model uses to generate outputs can be too complex to inspect or may not correspond to any practical human understanding of how the relevant domain works. Perfect transparency into source code and training data does not guarantee accountability in any meaningful sense.
This collision between administrative law’s demands and AI’s opacity is the central problem. But the more concrete problem is that existing doctrine does not even reach the collision in most cases.
Start with enforcement. The Supreme Court held in Heckler v. Chaney that an agency’s decision whether to initiate enforcement is presumptively insulated from judicial review. The Court recognized enforcement as a core executive function committed to agency discretion, involving a complicated balancing of factors that courts are poorly positioned to second-guess. The doctrine is grounded in two ideas: comparative expertise, meaning agencies know better than judges how to allocate scarce enforcement resources, and indeterminacy, meaning the grounds for individual enforcement decisions often cannot be reconstructed with any reliability.
Neither idea translates cleanly to algorithmic enforcement. A machine learning model that flags cases based on a specified objective function may be more tractable than a line-level prosecutor’s all-things-considered judgment. That tractability could, in theory, provide the focal point for judicial review that Chaney found missing. But an NLP-based model with hundreds of features and complex interactions may be more opaque than even the most informal human judgment. The result is paradoxical: the more intelligible a model is, the more reviewable it becomes. The more opaque it is, the more insulated it becomes from oversight.
The practical barriers compound this. An enforcement target that believes an algorithmic tool wrongly flagged it cannot challenge that decision on an interlocutory basis. Under Standard Oil of California v. FTC, an agency’s decision to initiate an enforcement action is non-final and therefore not immediately challengeable. The target must wait until the enforcement action reaches a conclusion. If the target loses on the merits, the challenge to the algorithm merges with the substantive liability question. If the target wins, Standard Oil’s rejection of litigation costs as legally cognizable injury makes it nearly impossible to obtain any remedy for the harm of having been investigated. Neither scenario produces systematic review of the algorithmic tool itself.
The non-enforcement side is no better. Regulatory beneficiaries who believe an agency’s algorithmic tools are under-enforcing can challenge those decisions only under narrow exceptions to Chaney’s presumption against reviewability. The strongest exception applies when Congress has mandated enforcement using specific standards. Federal statutes meeting that requirement are rare. The other exceptions, including abdication of statutory responsibilities and policy statements that self-cabin agency discretion, have been interpreted narrowly by courts and provide only irregular prospects of getting past the presumption.
Move to adjudication, where reviewability poses fewer formal barriers. The chances of meaningful review of specific algorithmic tools remain slim. QDD beneficiaries have no incentive to challenge the methodology that granted them benefits. Claimants who were not selected for QDD can challenge the ultimate outcome, but their challenge to the algorithm will likely merge with the merits of the benefits determination. Courts will apply a harmless error analysis: if you lost your underlying claim, did the algorithm’s failure to expedite your case actually harm you? The answer, in most cases, will be that back pay would ultimately be awarded, that the algorithm itself did not determine eligibility, and that judicial scrutiny of the technical methodology is not warranted.
Procedural due process offers little additional traction. The Mathews v. Eldridge framework requires courts to balance the private interest, the government interest, and the probable value of additional process. The earlier receipt of benefits through QDD, weighed against the cost of allowing all SSA applicants to probe the validity of a machine learning model in hearings, does not obviously favor additional process. And even if it did, the framework is designed to evaluate individual case outcomes, not systemic algorithmic errors that affect thousands of cases simultaneously.
The notice-and-comment process is available as an ex ante accountability mechanism, but its applicability is uncertain and its coverage is uneven. Agencies must subject legislative rules to notice and comment. The question of what counts as legislative, as opposed to interpretive or procedural, has generated case law that courts have described as tenuous, baffling, and enshrouded in considerable smog. The SSA’s QDD process did go through notice and comment because it required amendment of existing procedural rules. But the notice provided only minimal technical detail about the algorithm: that it would score claims based on medical history, treatment protocols, and symptoms, and that cases above a certain probability threshold would qualify. Whether that level of disclosure was legally sufficient is unclear.
More fundamentally, requiring notice and comment for all algorithmic tools would be counterproductive. The clustering algorithm SSA uses to reorder case processing within a judge’s docket falls squarely within the APA’s exemption for rules of internal agency organization. Forcing that tool through notice and comment would impose costs without producing accountability benefits. The real problem is not a shortage of regulatory procedures. It is that existing procedures were designed for a different type of agency action and do not map well onto how machine learning systems actually work.
The informational challenges that would face any reviewing body, whether a court or a comment-receiving agency, go well beyond technical opacity. When agencies contract with third parties to build algorithmic tools, the code may be protected by trade secrecy or copyright. When agencies build tools in-house, the code may be exempt from FOIA under law enforcement or trade secrecy exemptions. Training data is often protected by the Privacy Act, FOIA exemptions, or both. In the SSA context, individual claimant data is protected under the Privacy Act. In the SEC context, data from prior investigations used to train supervised learning models is likely exempt from FOIA disclosure as law enforcement sensitive.
Even where code and data are accessible, the information may be insufficient. A facial recognition model can appear flawless in code while producing racially biased outputs because of underrepresentation in training data. The dynamic nature of machine learning means that the model reviewed during a notice-and-comment period may already be substantially different by the time it is deployed. Models that use active learning or reinforcement learning update continuously. The APA process captures a static snapshot of a system that is designed to change.
The adversarial dimension adds another layer. Well-resourced regulated parties can probe government models and develop techniques to evade detection. Deep learning models can be fooled by adding imperceptible noise to an image or by inserting text that is meaningless to a human reader but shifts a model’s classification. Major investment banks have the quantitative capacity to build adversarial models against the SEC’s enforcement tools. Sophisticated applicants could, in theory, structure trademark filings to evade the Patent and Trademark Office’s image similarity search. Inadvertent underenforcement caused by adversarial manipulation may never be detected because the agency is not looking for it.
The result of this doctrinal and informational landscape is that algorithmic decision making at the SSA and SEC operates largely outside the accountability mechanisms that administrative law provides. Courts rarely review it. Notice and comment rarely covers it. When review does occur, it is triggered by individual case challenges that cannot address systemic problems. The agencies most likely to adopt AI are, in some cases, the agencies that have strategically designed their programs to minimize exposure to judicial scrutiny.
Three Approaches to Accountability
Acknowledging the failure of existing doctrine, there are three possible responses. Each has real value. Each has real limits.
Retrofitting the Administrative Procedure Act
The first approach is to adapt existing APA mechanisms to cover algorithmic decision making more effectively.
On notice and comment, courts and agencies need clearer guidance on when AI adoption triggers the legislative rule requirement. Several factors matter. The more humans remain genuinely in the decision loop, the less case there is for notice and comment. But human review cannot be a procedural formality. If caseloads, production quotas, or institutional pressure cause adjudicators to functionally ratify algorithmic recommendations without meaningful independent review, the system has displaced human discretion even if the forms say otherwise. The probability threshold that determines when a case is flagged for human review is a concrete indicator: a threshold near zero means all cases get human review; a threshold near one means the algorithm effectively decides. Setting that threshold is a policy choice with distributive consequences, and it is exactly the kind of choice that public participation can improve.
On distributive consequences: when algorithmic adoption creates distinct demographic effects, there is stronger case for notice and comment even at the development stage. QDD expedites benefits for applicants who file electronically. Paper filers, who skew older, less technologically connected, and in some demographics more rural, do not get the same advantage. That disparity is a policy question that extends beyond the technical parameters of the model.
On enforcement specifically, the case for notice and comment is weaker. The same reasons FOIA exempts enforcement data from public disclosure apply to algorithmic enforcement tools. Exposing the specific features and thresholds of the SEC’s ARTEMIS or ATLAS systems would allow regulated parties to design around them. The accountability value of notice and comment is lower in enforcement than in adjudication, and the cost in terms of tool effectiveness is higher.
On reviewability, the enforcement context is where congressional action or judicial reinterpretation of Chaney would do the most work. Congress could modify the presumption against enforcement review where agencies are using systematic algorithmic tools to prioritize targets. Courts could find that the mechanical, rule-bound nature of algorithmic enforcement tools provides the focal point for review that Chaney found missing in discretionary human judgment. In adjudication, channeled jurisdiction should be broadened to allow claimants to seek remedies that address systematic algorithmic error, not just the outcome of their individual case.
These modifications would expand accountability. They would also impose costs. Judicialization of agency decision making creates delays, diverts resources from regulatory work, and disrupts agency priority setting. More notice and comment does not automatically produce better notice and comment. Courts reviewing machine learning systems face genuine capacity limitations that do not disappear simply because jurisdiction is available.
An Algorithmic Oversight Board
The second approach is an institutional solution: a dedicated oversight board within each agency, or shared across structurally similar agencies, with responsibility for reviewing AI adoption and use.
Such a board would provide both ex ante and ex post oversight without the full costs of rulemaking or judicial challenge. It would include senior agency staff responsible for developing use cases, the agency’s Evaluation Officer or Chief Data Officer as mandated under the Foundations for Evidence-Based Policymaking Act, outside academics, stakeholder representatives such as disability rights groups or industry organizations, and representatives from peer agencies with comparable programs.
The board’s functions would be to provide input on a strategic AI plan, assess whether AI deployment complies with applicable law and policy including due process and antidiscrimination requirements, review operational details of deployed tools, and recommend revisions or decommissioning where warranted.
The potential benefits are concrete. A board could assess distributive impacts before they become legal problems. It could facilitate knowledge sharing across agencies that are working on similar problems in isolation. The SSA, the Board of Veterans Appeals, the Executive Office for Immigration Review, and the Office of Medicare Hearings and Appeals all face structurally identical challenges in mass adjudication. The SEC, the IRS, and the EPA are all building enforcement targeting tools from similar data sources. An oversight board shared across these agencies could capture economies of scale in both governance and innovation.
A board could also address adversarial learning systematically. If sophisticated regulated parties are developing techniques to fool government models, an institutional body with technical capacity is better positioned to detect and respond to this than individual program offices operating independently.
The costs are real. More oversight structure is not automatically better governance. If resource constraints are the primary barrier to AI development in federal agencies, diverting additional staff time and budget to an oversight board may slow adoption of tools that would improve outcomes. A board that lacks access to operational details of the tools it is supposed to oversee will produce recommendations that are disconnected from how the systems actually function. And agencies may resist creating boards precisely because external review might surface problems that program offices would prefer to manage quietly.
Prospective Benchmarking
The third approach is the most technically rigorous and the most directly tied to how machine learning systems should be evaluated: prospective benchmarking.
The core idea is straightforward. When an agency adopts an AI decision-making tool, it reserves a random sample of cases for processing under the conventional, non-algorithmic approach. That human-processed sample provides a comparison group against which the AI-assisted decisions can be evaluated. The benchmarking is prospective, meaning it runs alongside the deployed system rather than looking backward at historical decisions. Retrospective evaluations face inference problems that prospective random holdout designs avoid.
The practical implementation varies by context but follows the same logic. At the SSA, the Insight system could be disabled for a random subset of cases, with outcomes compared against those processed with Insight active. At the SEC, investigators could be required to fully work up a random sample of cases without access to risk scores, with the results compared to algorithmically assisted decisions. At the Patent and Trademark Office, the trademark similarity search tool could be disabled for a random set of applications.
This approach enables agencies, courts, and the public to assess what the AI tool is actually doing. Is it more accurate than human review? Is it faster? Does it produce disparate outcomes for identifiable demographic groups? Does it systematically miss certain types of errors? Does it create automation bias that makes nominally human review functionally meaningless? These questions cannot be answered from inside the algorithm. They can only be answered by comparing AI-assisted decisions to genuine human decisions on comparable cases.
Benchmarking provides several specific accountability benefits that other approaches cannot.
It validates the tool in real time rather than relying on agency assertions of improvement. The SSA’s claims about the clustering algorithm’s productivity gains were based on voluntary adoption data. Branches that chose to adopt the system may have differed from those that did not in ways that explain the performance differences entirely. A randomized holdout design eliminates this confound. The productivity gain is real only if randomly assigned cases show the same improvement.
It catches temporal and domain drift. Machine learning models are trained on historical data. When the underlying population of cases, claimants, or regulated parties changes, the model’s predictions may degrade without anyone noticing. Ongoing benchmark data provides a continuous signal about whether the model is still performing as designed. If the benchmark cases start diverging from the AI-assisted cases in ways that cannot be explained by the model’s intended function, that is evidence of drift that warrants investigation.
It preserves institutional capacity. One underappreciated risk of AI adoption is that agencies lose the human expertise required to process cases without the tool. If the SSA becomes entirely dependent on Insight to catch errors, and Insight is found to have a systematic flaw, the agency may lack the staff experience to return to manual review. Maintaining a benchmarking team ensures that human expertise does not atrophy.
It generates the information needed to update models. Benchmarked human decisions provide labeled data that can be used to retrain and improve algorithmic systems. If human reviewers in the benchmark group identify errors that the AI system missed, those cases can be used to improve the model’s training data. If adversarial learning by regulated parties has shifted the distribution of inputs in ways that are causing systematic misclassification, benchmark data will reveal the gap.
It provides an accountability mechanism that works even when agencies use proprietary contractor-built tools. An agency that contracts with a third party to build an enforcement targeting tool may not have access to the source code or training data. Benchmarking does not require access to the model’s internals. It requires only that a random sample of cases be processed without the model and the outcomes compared.
The limitations deserve acknowledgment. If human reviewers in the benchmark group have been trained on an AI system before participating in the benchmark, their manual decisions may not represent genuine pre-AI human judgment. They may focus on the specific error types that the AI system is designed to catch, missing errors outside the model’s scope. This risk is reduced by involving reviewers with substantial experience from before the AI system was adopted and by conducting reviews in teams rather than individually.
Mandatory human review of benchmark cases may delay processing for those cases. If the AI system is genuinely better than human review, the benchmark group bears a cost. This cost is the price of accountability. The alternative is deploying systems at scale without reliable evidence that they work as intended.
Human reviewers may have preferences about AI adoption that affect their benchmark performance. Reviewers who fear job displacement may over-invest in finding errors that the AI system misses. Reviewers who support AI adoption may unconsciously mirror what they expect the system would produce. Neither bias is easily eliminated. Team-based review, where multiple reviewers must reach a common assessment, reduces the risk that individual bias distorts the benchmark.
Despite these limitations, benchmarking is the only proposal that directly addresses the core epistemic problem: we do not know what these systems are actually doing in the field, and we cannot know without a genuine comparison group. Notice and comment provides input before deployment. Oversight boards provide review of documented outputs. Neither provides the controlled comparison necessary to measure impact. Benchmarking does.
How Benchmarking Gets Implemented
There are four paths to requiring prospective benchmarking.
Congress can mandate it by statute. A statutory benchmarking requirement would apply uniformly across agencies, provide a clear legal basis for the obligation, and allow Congress to specify minimum standards for how benchmarks are conducted, how results are reported, and what triggers a requirement to revise or decommission a tool. Congress could also attach judicial deference benefits to agencies that comply with benchmarking requirements, creating an incentive for voluntary adoption.
Courts can find that agency decisions made using AI tools without benchmarking are arbitrary and capricious under APA review. This path requires getting past the reviewability barriers documented earlier, which remain substantial in enforcement and uneven in adjudication. But where review is available, a court could reasonably hold that an agency claiming performance improvements from an algorithmic tool, while conducting no rigorous validation of those claims, has failed the APA’s reasoned decision-making requirement.
The President can mandate benchmarking by executive order. An executive order could require all federal agencies adopting machine learning tools for core governmental functions to implement prospective benchmarking as a condition of deployment. This approach does not require congressional action and can be implemented quickly. It is also reversible by a subsequent administration and may be interpreted narrowly by agencies with incentives to minimize compliance costs.
Agencies can adopt benchmarking voluntarily. Some agencies already conduct internal audits and quality reviews that share structural features with benchmarking. The Government Performance and Results Act requires agencies to measure performance against objectives. Benchmarking fits within this framework and requires no new legal authority. The practical barrier is incentive structure. Agencies have strong incentives to report that their AI tools are working. They have weak incentives to commission rigorous tests that might reveal otherwise.
Many implementation details remain to be resolved. How large must the benchmark sample be to provide statistically reliable comparisons? At what intervals should benchmarking be repeated beyond initial deployment? How should results be reported and to whom? Who makes the decision to revise or decommission a tool based on benchmark results? These questions do not have universal answers. They depend on the specific tool, the volume of cases, the stakes of individual decisions, and the costs of error in both directions. Best practices will need to develop through experience, ideally coordinated across agencies facing similar problems.
What Is Actually at Stake
The SSA processes millions of disability claims every year. The people who file those claims are, by definition, individuals whose medical conditions have prevented them from working. They are waiting, often for years, for a determination that will affect whether they can pay rent, cover medical costs, or provide for their families. An algorithmic tool that systematically advantages some claimants over others, or that converts ALJ review into the rubber stamping of machine recommendations, is not an abstract due process concern. It affects real outcomes for people who have no fallback.
The SEC enforces the securities laws against investment advisors who manage other people’s money. Being flagged as high risk by an algorithmic tool triggers examination scrutiny. Being referred to the enforcement division triggers investigation. Being investigated carries costs: legal fees, reputational damage, operational disruption. Those costs fall whether or not the underlying algorithmic classification was correct. If well-resourced institutions can reverse-engineer the tools and adjust their disclosures to avoid detection while less sophisticated actors cannot, algorithmic enforcement produces a systematic advantage for those who already have the most resources.
The broader question is whether AI adoption in the administrative state will improve how government exercises redistributive and coercive power, or whether it will replicate and amplify existing failures while making them harder to detect and correct.
That question does not get answered by arguments about AI’s potential. It gets answered by what agencies actually build, how those systems actually perform in the field, and what mechanisms exist to identify and correct failures before they compound.
The administrative law framework inherited from the APA era was designed for a different type of agency action. It assumes that agency decisions are made by identifiable human beings, that the reasoning behind those decisions can be articulated and reviewed, and that errors in individual cases represent the failure of individual decision makers that can be corrected through individual case review.
None of those assumptions hold for machine learning systems operating at scale. The reasoning cannot always be articulated. Individual case review cannot surface systematic errors. The decision makers are not individual human beings but statistical models trained on historical data that may not represent the population being governed.
Adapting administrative law to this reality requires more than procedural adjustments. It requires building into the governance of AI systems the same empirical discipline that good machine learning requires: test your system against a genuine comparison group, measure what it actually does rather than what you intend it to do, and build in the capacity to detect and correct failures before they become entrenched.
Benchmarking is not a comprehensive solution. An oversight board is not a comprehensive solution. Better notice-and-comment guidance is not a comprehensive solution. No single mechanism is. But benchmarking provides something the other approaches do not: direct empirical evidence about what AI systems are doing in practice, generated in a form that agencies, courts, and the public can actually use.
The Sunnyvale comparison returns. That city’s performance measurement system failed because officials could game the measures, change the weights, and eventually abandon the framework when it produced inconvenient results. The new algorithmic governance tools face the same failure mode. An agency that adopts an AI tool, reports internal improvements, declines to benchmark against a genuine human baseline, and adjusts the model whenever scrutiny increases is running the same playbook. The technology is more sophisticated. The accountability gap is the same.
The stakes are higher than they were in 1993. The tools being deployed now do not measure internal performance metrics. They decide disability claims and initiate securities enforcement actions. They carry the coercive power of the federal government. They affect people who cannot opt out, cannot reverse-engineer the model, and in many cases cannot afford the legal representation required to challenge the outcome.
Administrative law developed, over decades, to ensure that this power is exercised with transparency and accountability to the people it affects. That requirement does not become less important because the decision is made by an algorithm. It becomes more important, because the algorithm operates at a scale that individual human decision makers cannot match, and because failures that would be caught and corrected in individual human judgment can propagate unchecked through thousands of cases before anyone notices.
The technology is moving faster than the doctrine. Closing that gap is not optional. It is the work of accountability in a constitutional republic where the government answers to the people, and rights do not disappear because a machine made the call.
Margin of the Law publishes constitutional analysis, civic research, and legal education for people who want to understand the system they actually live in. Read the Full Constitutional Analysis Library at marginofthelaw.com.
© 2026 – MK3 Law Group
For republication or citation, please credit this article with link attribution to marginofthelaw.com.