Beyond the Black Box: Conditions for Credible Use of Artificial Intelligence in Verification

Across almost all technical and administrative domains, there is a clear trend toward the increasing use of artificial intelligence (AI). Data volumes are growing, sensors are proliferating, and political decision-making is increasingly expected to be informed by continuous monitoring and rapid analysis. Arms control, disarmament, and non-proliferation verification are not exempt from this development. Not every verification procedure requires AI support, but it can help address some of the challenges that verification regimes encounter and can serve as an enabling factor for the deployment of modern technologies. Some of these challenges include an ever-growing number of facilities needing to be safeguarded by a limited human workforce, the increasing availability of sensors for verification purposes, and the accelerating pace of geopolitical developments requiring fast and reliable compliance assessments. Here, AI is particularly attractive as it promises to support exactly those tasks that are resource intensive for humans: screening large amounts of sensor data, detecting anomalies, aggregating heterogeneous information streams, and accelerating otherwise lengthy verification procedures to make verification more efficient, responsive, and potentially more objective.¹

But verification is a highly political endeavor. Introducing AI into its processes raises questions about the explainability of the data processing, the credibility of its outcomes, the development and negotiation of the underlying algorithms, the selection and balance of the data used to train the AI, and even potentially unequal access of different participating actors to advanced models or computing infrastructure—all of which will be discussed in the following sections.

The Black Box Problem Reconsidered

The opacity of AI systems, in particular, seems to contradict the prerequisites for effective verification mechanisms. This so-called “black box” character of AI² has two closely interconnected dimensions: limited transparency and traceability of the system’s internal workings and, consequently, limited explainability of the outcomes it produces. This is rooted in the logic of machine learning methods, which are composed of large numbers of interconnected mathematical operations. In practice, users may receive a classification or recommendation without being able to directly identify which specific characteristics of the input data, or which effects of the training data, led to the result. While the inner workings of simple neural networks for classification and regression tasks can be explored reasonably well with explainable AI (XAI) techniques,³ this process is much more intricate for large language models (LLMs) and generative AI systems.⁴ These AI techniques enable the processing of complex textual information corpora and are therefore well suited to verification tasks for which human analysis was previously considered indispensable. However, the algorithms used are stochastic in nature, meaning that data processing is not mathematically deterministic and identical inputs may not necessarily produce identical outputs. These characteristics are not merely technical inconveniences. They affect the core function of verification, which depends on processes capable of producing results that are considered reliable and credible. Artificial intelligence thus enters the verification debate not only as a promising tool, but also as a potential new source of contestation, with actors seeking to exploit its limitations to question verification findings or even justify noncompliance by challenging the reliability of the verification system itself.

Upon closer inspection, however, the picture is more nuanced. Verification usually involves balancing transparency against the protection of sensitive information. The more intrusive or sensitive the verification provisions, the more difficult it becomes to strike this balance. For example, national technical means were accepted earlier in nuclear arms control than on-site inspections involving physical measurements. States seek to avoid disclosing more than is strictly necessary for verification purposes, while still providing sufficient evidence to establish compliance. From this perspective, opacity is not entirely new. Even traditional verification technologies are often complex and not fully transparent to all parties. Consider nuclear warhead verification. High-resolution spectra of emitted particles could confirm the presence of a warhead and rule out a hoax object. However, these spectra would also reveal sensitive design information, rendering their use politically unacceptable. While some approaches try to avoid gathering any sensitive information at all,⁵ others apply what are referred to as “information barriers”,⁶ which intentionally reduce the resolution of the measured data while still preserving enough information for the verification task.

The mere fact that an AI system is not perfectly transparent therefore does not automatically disqualify it. Likewise, determinism in data analysis should not be overestimated. Deterministic systems can also systematically produce misleading results if they rely on inappropriate assumptions or biased data. In addition, traditional data analysis can also suffer from nondeterminism due to statistical fluctuations. Provided that systems with probabilistic elements can be adequately characterized, repeatedly tested, and validated to ensure consistent output behavior, nondeterministic AI is viable for verification.

For verification purposes, the politically relevant category is therefore not determinism but credibility, ensuring that participating actors perceive the system as producing results that cannot be easily dismissed as arbitrary, manipulated, or one-sided.

Conditions for Credible Use of AI in ­Verification

Given this shift away from the necessity of an explainable AI system, the question is which technical and institutional requirements AI systems used in verification must satisfy for parties to regard the resulting assessments as credible, even in a limited-trust environment.

First, AI systems must be designed to protect sensitive information at all stages of the AI life cycle. Training modern AI models requires large volumes of data which, in an arms control context, may in some cases be open source, but will often originate from states, inspectors, and international organizations. Such data can contain information that extends beyond the scope of the treaty concerned and may reveal military capabilities, operational procedures, or technical characteristics that states are unwilling to disclose. Artificial intelligence models used for verification should therefore be developed according to the principle of data minimization. The technical parameters for this should ideally be developed and assessed by dedicated expert groups, thereby reducing the risk that highly technical issues become obstacles in political negotiations. Training data should be limited to the information necessary for the verification task, while remaining sufficiently informative to ensure that the AI’s capabilities are not constrained. Where the boundary should be drawn here may not be clear-cut and will depend on both technical requirements and political circumstances. Rather than seeking to provide broad analytical capabilities, model architectures should be designed to answer narrowly defined verification questions. As research has shown that data can be extracted from trained models⁷ despite safeguard measures, such measures must ensure that generated outputs do not exceed the agreed verification mandate. This design principle is especially critical with AI being such a fast-paced technology, which continually develops new capabilities as well as new vulnerabilities and attack vectors.

Second, the integrity of both data and models must be guaranteed. Verification ultimately depends on evidence. If the integrity of the underlying information can be called into question, the credibility of the entire process collapses. This challenge extends beyond protecting sensor data against tampering, as AI systems introduce additional attack surfaces through which adversaries may attempt to manipulate training datasets, inject misleading information into operational data streams, or modify model parameters—techniques commonly referred to as data or model poisoning.⁸ Consequently, verification architectures require strong mechanisms for ensuring provenance and traceability. Data streams should be cryptographically protected, stored in tamper-evident systems, and accompanied by verifiable chains of custody. Equally importantly, model versions and software updates must be documented and auditable, allowing changes to be tracked and reviewed. Secure data handling procedures already employed by organizations such as the International Atomic Energy Agency (IAEA) offer useful models that could be extended to AI-supported verification environments.

Third, reproducibility must become a central design principle, as it enables actors to test whether the model behaves as claimed, assess whether its limitations are properly understood, and independently reevaluate results. To achieve this, participating actors need to access source code and algorithms, detailed documentation of model architectures, hyperparameters, training data and procedures, preprocessing pipelines, and validation methods, so that they can recreate the training process under agreed conditions and obtain models with comparable behavior and performance characteristics. While exact bit-for-bit replication is often unrealistic given the stochastic elements of machine learning, comprehensive documentation and sharing of a specific technological setup can facilitate comparable outputs and statistically equivalent performance.

Fourth, verification systems should avoid excessive dependence on centralized infrastructures. Traditional verification mechanisms often rely on trusted international organizations such as the IAEA or the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO) to collect, process, and evaluate information. While such organizations play an indispensable role, the introduction of AI raises new concerns about covert manipulation of software, models, or data—whether through external cyberattacks or deliberate actions by participating actors. To mitigate this risk, participating actors could operate identical or functionally equivalent versions of the verification software and independently process the same datasets, thereby creating a decentralized verification architecture with equivalent analytical capabilities. Results could then be compared across multiple implementations and institutions and tested to see if different actors independently reach similar conclusions. This would, in turn, strengthen confidence in the verification process.

Fifth, transparency must be understood as a practical rather than absolute requirement. Full transparency is neither politically realistic nor necessarily desirable, and it would probably undermine participating actors’ intentions and willingness to compromise. The degree of transparency should therefore be sufficient to enable actors to understand what a system is designed to do, which data it uses, how its performance was evaluated, and where its limitations lie. This does not necessarily require unrestricted public access to all models and datasets. Approaches could include open-source model architectures, jointly developed reference implementations or curated training datasets, independent technical audits or certification procedures conducted by designated expert bodies, joint testing exercises, and benchmarking datasets.

Lastly, the hardware environment deserves greater attention than it typically receives in AI governance debates. Advanced AI increasingly relies on specialized computing infrastructure, large-scale data centers, and access to high-performance processors. These resources are distributed unevenly across the international system. Such asymmetries create political as well as technical challenges.⁹ Verification systems that rely on infrastructure controlled by only one or a small number of actors may create concerns about dependence, unequal influence, and hidden capabilities. Capability-building measures, together with support for or jointly managed computational facilities, can help allay concerns about technological asymmetries, thereby becoming important confidence-building measures and paving the way toward effective verification regimes.

Taken together, these requirements suggest that the main issue is not whether AI is perfectly explainable. Rather, the challenge lies in embedding AI systems in verification regimes in a manner that preserves reproducibility, integrity, transparency, and political acceptability. If these conditions are met, even complex AI systems may become credible verification tools. If they are neglected, however, even the most sophisticated AI systems may ultimately fail, because their results will not be accepted by the actors they are intended to convince.

Governance Trade-Offs

These requirements are demanding and create governance dilemmas of their own. States have incentives to shape datasets and model designs in ways that protect their own interests, while detailed knowledge of verification models could enable adversarial behavior or facilitate attempts to evade detection. Arms control agreements have long entailed negotiations over technical specifications, inspection procedures, and sensor capabilities. Similar debates surrounded issues such as, for example, the scope and accuracy of sensors used under the Open Skies Treaty.¹⁰ Artificial intelligence should therefore not be viewed as a fundamentally different category of verification technology. Rather, it recasts familiar verification dilemmas in a new technical form. The question for policymakers and practitioners is therefore not whether AI should be used in verification, but under what conditions it is likely to strengthen rather than weaken the credibility of verification regimes.

The central challenge here is not the existence of a “black box” as such, but ensuring that AI systems are designed and governed in a way that supports credibility in an environment characterized by limited trust. Reproducibility, integrity, transparency, decentralization, and robust security are the key prerequisites for the political acceptance of AI-supported verification. When these conditions are met, even complex AI systems can become credible and effective tools for arms control verification.