Peter Cihon
Moritz J Kleinaltenkamp
Jonas Schuett
Seth D Baum

AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries

Certification can be defined as the attestation that a product, process, person, or organization meets specified criteria (paraphrased from  [6] ). Certification involves an assessment of the entity to be certified, often, but not necessarily, by a trusted third party. If the assessment determines that the entity meets the specified criteria, then the entity is certified accordingly. Thus, the certification process reveals information about the inner workings of the entity that can be provided to external stakeholders, reducing information asymmetries between insiders and outsiders  [7] . In doing so, certification incentivizes insiders to adhere to higher standards because they can get credit for it if they do and get caught if they do not. In short, certification is a governance tool for advancing transparency and incentivizing insiders to meet specified criteria. Those criteria could include voluntary AI ethics principles or mandatory regulatory requirements. Certification is already in wide use in many sectors  [8] ,  [9]  and is starting to appear for AI. Certification includes such disparate phenomena as the U.S. EnergyStar program for consumer appliance energy efficiency, ISO 9001 for quality management systems in global supply chains, and the mandatory "CE mark" for various products sold in the EU. Certification programs for AI systems have been proposed or are currently being developed by the European Commission  [10] , the IEEE  [11] , and by the Chief Scientist of Australia  [12] ; one has already launched from the government of Malta  [13] . Education certification programs include the Queen's University executive education program on Principles of AI Implementation  [14]  and the Finnish civic education program Elements of AI  [15] . Additionally, a coalition of Danish organizations  [16]  is developing an IT security certification for organizations that includes AI-related criteria. These various programs speak to the diverse forms that certification programs can take, including public and private, voluntary and compulsory, and for the AI systems themselves as well as the people and organizations involved with them. This article presents an overview of certification for advancing ethical AI. It provides background on AI certification, drawing on management literature (Section II), surveys current AI certification programs and proposals for new programs (Section III), and discusses prospects for certification to address issues raised by future AI technologies (Section IV). Section V concludes this article.
 II. BACKGROUND This section reviews management literature on the merits of certification as it relates to the governance of AI. This literature typically focuses on the market behavior of corporations and their customers. However, certification is also relevant to nonmarket settings. For example, AI systems developed by nonprofit and/or open-source teams could seek certification to build user confidence, even if users are not charged for using the systems. Or, governments might seek international certification of their military AI systems to demonstrate compliance with international humanitarian law or other standards. Thus, while certification is important for the corporate governance of AI (on which, see  [17] ), it is also more widely applicable. First, some terminology. The object of certification is the entity to be certified. It may be a product, process, individual, or organization. The certifier is the actor who does the certifying. The assessor is the actor who assesses whether or not an object of certification meets the specified criteria prior to certification. The assessor and certifier can be one and the same or they can be two separate actors, such as in the case of certification programs that use third-party assessment. The assessment or conformity assessment is the evaluation of whether the object meets the specified criteria; this can vary in stringency, from a review of application materials to an audit of systems or processes. The applicant is the actor who controls the object of certification and seeks to obtain certification when it is voluntary. The applicant and object of certification can be the same, such as in the case of skills certifications granted to individuals. Alternatively, the applicant can be the entity who oversees, manages, develops, or otherwise controls the object of certification. Finally, the audience is the group of people who will be informed by the certification. For example, consider an AI system developed by a corporation, certified by a government agency, assessed by an independent consulting firm, to inform consumer purchasing decisions. Here, the object is the AI system, the certifier is the government agency, the assessor is the consulting firm, the applicant is the corporation, and the audience is consumers. In some cases, the applicant may also simultaneously be the certifier; this is known as self-certification. Self-certification creates a conflict of interest, though this can sometimes be addressed via measures such as audits and liability regimes. Two additional sets of terms help discuss the governance potential of certification programs. First, symbol-substance coupling refers to alignment between the statement that an object meets the certification criteria (the symbol) and that object actually meeting those criteria (the substance). Likewise, symbol-substance decoupling is when a certification inaccurately characterizes its object  [18] -  [20] . Second, means-ends coupling refers to alignment between the means of the certification program (i.e., the specific criteria to which objects are being held) and the ends underlying its design  [19] ,  [21] . Means-ends decoupling is when a certification program fails to advance its intended goals. Symbolsubstance decoupling and means-end decoupling are two failure modes of certification programs.
 A. Reducing Information Asymmetries As noted above, a primary role of certification is to reduce information asymmetries. Information asymmetries raise two issues: the "selection problem" of identifying desirable suppliers of a good or service, and the "monitoring problem" of evaluating whether the supplier is meeting specified agreements  [22] . Both problems create space for rent-seeking behavior, in which the supplier extracts greater wealth from other parties without adding social value  [23] . Information asymmetries are acute in AI systems because the systems are often complex and opaque and users typically lack the data and expertise necessary to understand them. Certification is especially well suited to addressing such information asymmetries that arise from "credence" and "Potemkin" attributes of goods and services  [24] . Credence attributes are those that can be identified by outside parties, but only as trends that emerge after repeat observations. Individual audience members may struggle to identify credence attributes on their own, but an organized third party, such as a certification program, can collect the data needed for a reliable characterization. Credence attributes of AI systems include whether a system is fair or unbiased, is explainable across a wide range of use cases, protects user privacy across a wide range of settings, and is safe and secure to rare and/or diverse threats. Potemkin attributes present even greater information asymmetry challenges: they cannot be identified at all by outside parties, and instead require insider information. An important type of Potemkin attribute is the process through which the good or service was made, such as whether an AI system was developed in accordance with labor, environmental, or ethical standards, or whether it was rigorously tested prior to deployment. Certification programs can address both credence and Potemkin attributes, as seen for example in the popular ISO 9001  [25]  certification for organization quality management and ISO 14001  [26]  certification for environmental management. By providing information about credence and Potemkin attributes, suppliers can create new signals to attract outside parties, including market customers  [27]  and others. Table I presents a summary overview of the key concepts underlying our understanding of certification, and how we apply them to the field of AI. A certification program reduces information asymmetries when it accurately characterizes the objects that it certifies. It must avoid false positives, in which an object is certified even though it fails to meet the specified criteria, and false negatives, in which an object meets the criteria but is denied certification. Accurate certification thus exhibits high symbolsubstance coupling. Research on certification has identified strategies for improving certification accuracy, i.e., improving symbolsubstance coupling. First, the objects to be certified should be assessed continuously, rather than only once or not at all  [7] ,  [28] -  [30] . This ensures that the information conveyed by the certification stays up to date. Second, assessments should be made by qualified independent assessors using their own evidence and not just documentation provided by the applicants  [24] ,  [31] ,  [32] . This is needed to protect against applicants misleading the assessor, especially to gain a false positive. These strategies are of further value for incentivizing good behavior by the object and/or applicant, as discussed in the section below. Indeed, the strategies were largely developed to promote behavior change. To achieve accuracy, AI certification programs will often need to parse technical details of the systems and their development. Evaluation methods required for accurate AI certification are at an early stage of development. The most mature area may be documentation for AI system training data and model characteristics  [33] -  [35] , which can help verify certification criteria associated with both credence and Potemkin attributes. Supply-chain and development documentation methods that draw on insights from other fields may help verify Potemkin-attribute criteria  [36] ,  [37] . Further, benchmark tests to assess AI system performance for specific tasks in specific settings  [38]  and more general behavioral settings for specific types of systems  [39]  can help verify credence-attribute criteria. However, methods for criteria verification in broad areas of fairness, transparency (explainability), safety, and security remain under development. Certification programs aiming to address these areas will need to update their criteria over time as the state-of-the-art advances.
 B. Incentivizing Change In reducing information asymmetries, certification can further serve to incentivize changes to applicant practices. For example, research has shown that corporations may be more motivated to achieve ethics standards if they can use certification to demonstrate their achievements to customers who value these achievements  [31] ,  [40] . Where there is demand for certification-e.g., from corporate customers who value ethics achievementscertification programs can succeed on a voluntary basis, with no government regulation  [9] . Government regulation can further boost voluntary programs via tort liability  [41]  or tax credits. Mandatory certification programs display higher compliance levels, though voluntary programs are sometimes found to be more cost effective  [42] . For a certification program to change applicant practices according to its specified criteria, it is important that it accurately characterizes these practices and their outputs. In particular, false positives permit applicants to pursue certification without changing their underlying practices: they can adopt the certification criteria symbolically but not substantively. Such symbolic adoption is of particular concern in AI. The recent popularity of AI ethics principles has prompted concerns about "ethics washing"  [43] ,  [44] , in which AI groups articulate ethics goals to bolster their reputation but do not act accordingly. An accurate certification regime could mitigate the problem of AI ethics washing. Prior research identifies several strategies to motivate substantive instead of symbolic participation in certification programs. First are strategies to improve certification accuracy as discussed above. Second, certification assessments should not be onerous on the applicants; otherwise, applicants that meet the assessment criteria may decline to participate in the program  [24] . Third, the cost to applicants of failing an assessment should be high (in terms of damaged reputation, lost market power, etc.), so as to incentivize participating applicants to meet the specified criteria  [22] ,  [24] ,  [27] ,  [31] . A successful certification program may need to promote its value so that other parties (e.g., customers) know to penalize applicants that fail certification assessments. Finally, certification programs should require applicants to integrate the values and norms underlying the criteria into their organizational identities and strategies; this makes substantive adoption more likely  [30] ,  [45] . A robust body of research suggests that properly constructed certification programs can couple symbol and substance to accurately characterize and induce changes to corporate practices. For example, studies of ISO 14001  [26]  have found a significant correlation between: 1) the ISO 14001 attestation and the presence of environmental management practices in corporations and 2) the presence of these practices in corporations and their improved environmental performance  [7] ,  [46] . Similar evidence exists for the ISO 9000 series on quality management practices  [47] ,  [49] . Ultimately, adherence to certification criteria depends on applicants' actions. Their actions can derive from characteristics of themselves in addition to characteristics of the certification program. Indeed, applicants sometimes engage in a mix of symbolic and substantive adoption even within the same certification program  [47] . This may be due to a lack of rational reflection or to internal disagreement within the applicant about the importance of the certification criteria  [45] . Additionally, some studies have found that firms with higher revenues tend to adopt certification more substantively because they have the resources to meet the certification criteria  [51] ,  [52] . Therefore, while it is important for certification programs to be well designed, the applicants retain an essential role.
 C. Ensuring Certification Advances the Right Goals The preceding section discussed how to ensure that certification criteria are in practice actually met, i.e., that certification is adopted substantively and not just symbolically. The current section covers the selection of the criteria themselves. These criteria should be designed such that if they are achieved, the certification program will deliver improvements on its underlying motivation: advancing ethics principles, making progress on societal issues, etc. In other words, the means of certification must be coupled to its ends. Means-ends decoupling can arise when developers of certification programs fail to treat the focal problem holistically, leading them to misidentify the causal relationships driving the issue at hand and prescribe the wrong action for a particular problem  [21] . This can spur at least three types of mistakes. First, certification may be inadequately customized for the circumstances it addresses. A common shortcoming of certification programs is the use of uniform, homogeneous prescriptions that neglect important local and time-specific factors  [21] . For example, AI systems required to use a certain dataset that performs well on demographic diversity could end up underperforming when even better datasets become available. Some of the most popular corporate certification programs make homogeneous prescriptions for corporations across the globe, though a context-dependent multiplicity of corporate practices may be necessary to address societal problems across environments. Certification criteria that are customized for particular circumstances may tend to perform better, though the development of such criteria may require more resources. Second, certification can cause unintended harms. Narrowly crafted certification criteria can neglect important causal relationships; when followed, they may make some things better but other things worse. For example, AI systems meeting certain performance standards may require additional computing power, which inadvertently harms the environment via increased energy consumption  [53] . This type of problem is also known as the "waterbed effect" because lying on a waterbed simply displaces the water: the problem "rises" again somewhere else  [21] ,  [54] . To avoid unintended harms, certification programs have to avoid overly simplistic prescriptions and instead embody and promote systemic thinking about the issues at hand  [21] . Third, certification criteria can be overly restrictive, inhibiting applicants from taking innovative actions that would better address the particular problem. One way to make actions more customized to complex causal relationships expressed in particular circumstances is to permit applicants to make decisions on a case-by-case basis. However, certification criteria often prescribe a rigid set of actions, thereby inhibiting applicants from identifying better solutions  [21] ,  [55] . Certification programs can mitigate this type of problem by "stimulating internalization," meaning that they de-emphasize rigid rules and instead emphasize changes to applicants' internal culture, such that applicants are likely to do the right thing as circumstances dictate  [21] . Related to these challenges is the question of whether a certification program addresses the right focal problem(s) to begin with. Exactly which problems should be focused on is ultimately a question of ethics. Opinions may vary on which issues are most important. For example, many AI experts are divided on whether the field should focus on issues raised by near-term or long-term AI  [56] . Certification programs may sometimes need to take sides on these sorts of divides, such that they "cannot please everyone," with some contending that the programs work on the wrong issues. In other cases, certification programs could emphasize issues that may seem appealing at first glance because of their broad scope, but become disagreeable when actors' interpretations of this broad scope begin to differ. To ensure that certification programs overcome these challenges and address their focal problems as holistically as possible, their developers should draw on the expertise of a wide range of stakeholders and carefully reconcile these stakeholders' differing perspectives.
 III. AI CERTIFICATION PROGRAMS AND PROPOSALS To identify existing AI certification programs, we conducted Internet searches, monitored social media, solicited input from colleagues, used our own prior knowledge, and examined references from documents about the programs and other relevant literature. These programs are not exhaustive, but are largely representative of the variation in AI-related certification.  1  We assessed the programs using publicly available documents. To fill in gaps in the documents, we conducted semi-structured interviews with representatives of certification program organizations. In total, we conducted three such interviews across seven identified AI certification programs. A) European Commission White Paper on Artificial Intelligence.
 B) IEEE Ethics Certification Program for Autonomous and Intelligence Systems (ECPAIS). C) Malta's AI Innovative Technology Arrangement (AI ITA). D) Turing Certification proposed by Australia's Chief Scientist. E) Queen's University's Principles of AI Implementation executive education. F) Finland's civics course Elements of AI. G) Danish labeling program for IT-security and responsible use of data. The seven programs classify into four categories: 1) selfcertification of AI systems (A); 2) third-party certification of AI systems (A-D); 3) third-party certification of individuals (E and F); and 4) third-party certification of organizations (D and G). Table  II  presents our ratings of these categories in terms of feasibility, symbol-substance coupling, and meansends coupling. The ratings are based on our analysis of the  1  Additional certification programs under development include AI Global's AI system certification  [57] , CertNexus's certified ethical emerging technologist program  [58] , and ForHumanity's compliance audit for AI systems and organizations  [86] . A number of universities offer AI ethics-related certifications or minors, including San Francisco State [59] and Carnegie Mellon  [60] . Consultancies offer algorithmic auditing services, which may confer a certification  [61] . Self-certification is widely used in many industries globally, including suppliers' declarations of conformity used in telecommunications and motor-vehicle manufacturing  [62] . For self-certification of AI systems, we rate certification programs in this category as having high feasibility because it requires little involvement from nonapplicant actors and limited back-and-forth between them and the applicants. The symbol-substance coupling is low because of its inherent conflict of interest and lack of strong enforcement and assessment. These problems can be attenuated via strong ex-post enforcement after harms occur via liability regimes. However, liability is problematic especially for high-risk applications where the initial harm could be catastrophic, and because legal liability adjudication poses uncertainties for AI developers  [63] . Finally, the means-ends coupling is medium because it encourages applicants to take ownership of the certification process and all that it represents, including by documenting their AI practices and thinking systemically about the corresponding ethical implications. Third-party product certification is used in numerous industries globally, including Underwriters Laboratories certifications for electrical and fire safety and the Common Criteria  [64]  for product cybersecurity certification. For thirdparty certification of AI systems, we rate certification programs in this category as having medium feasibility because AI systems are a relatively well-defined object to assess and certify, but they are also a novel one for which prior experience is of limited use. The need for qualified third-party organizations to offer assessment adds additional complications, including questions around costs and accountability  [5, p. 12] . Symbolsubstance coupling is medium because, on the one hand, third parties can assess certification claims and demand for certified systems could drive substantive adoption, but on the other hand, actual implementation can fall short on both meaningful assessment and enforcement. Finally, means-ends coupling is medium because of the potential for, but difficulty of, successfully assessing each AI system in its development and deployment context. Third-party certification of individuals is common in a number of professions, including among engineers, lawyers, and actuaries, as well as programs like the Certified Information Privacy Professional credential  [65] . We rate AI-related certification programs in this category as having medium feasibility. A core variable is whether certification criteria include matters related to moral factors in addition to the usual criteria of technical knowledge and skills.  2  Assessment of moral standards is important but less feasible. Moral character is not readily testable, and educational institutions may have less financial incentive to do so. The symbol-substance coupling is low because the substance of certification criteria is dynamic. Methods in explainability, fairness, safety, etc., are active research topics; certificates attesting to knowledge of them can become quickly outdated. This problem could be addressed via ongoing audits or education requirements comparable to those sometimes required for lawyers, but current AI certification programs lack such measures. Finally, the meansends coupling is low because, absent robust and dynamic moral standards, certification is unlikely to guarantee that the individuals will achieve sound performance on AI issues. Third-party certification of organizations include the U.K. Cyber Essentials  [66]  cybersecurity company certification and the FairTrade  [67]  licensing contract. We rate AI-related certification programs in this category as having medium feasibility because it can draw on a preexisting body of experience and knowledge in corporate governance and related fields, but it still requires AI-specific innovations. The symbol-substance coupling is high because it can use robust preexisting methods like audits and data security management. Organizations are often familiar with these methods and understand how to respond with substantive activity. Their familiarity could also be used to game the system, though auditors are themselves familiar with such tactics. Finally, the means-ends coupling is medium because assessments can support the adoption of a "systemic mindset" but certification criteria may not be easily customized for the context of a particular organization.
 A. European Commission White Paper on AI The European Commission White Paper on AI  [10]  proposes separate certification programs for low-risk and high-risk applications. Both programs would draw on prior certification programs, in particular, those of the European Economic Area "CE marking" rules and the 2019 EU Cybersecurity Act. Both programs would also use similar certification criteria, such as the EC's AI Ethics Guidelines for Trustworthy AI  [68] . If implemented, the programs would be important in their own right and could further be influential for AI certification beyond the single market, as is often the case for EU policies  [69] . For low-risk applications, we interpret the White Paper as calling for voluntary self-certification. This is clear from the White Paper's call for following the precedents of CE marking and the Cybersecurity Act, both of which use self-certification for low-risk applications. There is some ambiguity because the White Paper also suggests for the low-risk program to use a "combination of ex ante and ex post enforcement"  [10, p. 24] . Ex ante enforcement would seem to preclude self-certification. We interpret a stronger significance to the two self-certification precedents and therefore believe that the White Paper intends the low-risk program to involve voluntary self-certification. AI applicants could attest to their system having met the criteria, and face ex post penalties if their system falls short. Applicant participation would be incentivized by consumer demand for certificates of AI trustworthiness  [10] . To be effective, the low-risk program would need to be carefully designed. First, it needs an enforcement mechanism with clear, strong penalties for applicants who falsely certify their AI systems as meeting the specified criteria  [70, p. 5] . Otherwise, there could be symbol-substance decoupling, with applicants rubber-stamping their systems as meeting the criteria. Fortunately, the program could build on the compliance infrastructure that already exists for CE markings. Second, the criteria need to capture the many aspects of AI trustworthiness as defined in the "Ethics guidelines for trustworthy AI"  [68] , which include, among other things, fairness, security, and accountability. The criteria further must promote the systemic mindset needed to advance this conception of trustworthiness. Otherwise, there could be means-ends decoupling, with the certification program not advancing its underlying goals. For high-risk applications, the White Paper clearly specifies mandatory third-party certification via what it refers to as "conformity assessment"  [10, p. 23] . Its procedures would be based on CE marking rules or the Cybersecurity Act  [10] , both of which require third-party verification for high-risk applications. The White Paper's program would use existing accredited auditing organizations ("notified bodies") of the Member States  [10, p. 25] . EU regulation would compel AI applicant adoption and compliance. To be effective, the high-risk program will also need several design elements. First, it needs to ensure that the certifying bodies have the technical expertise and capacity needed to assess AI systems. Using the existing notified bodies makes the program easier to implement, but the bodies currently lack expertise and capacity on AI. Unless this situation is improved, the assessments could be inaccurate, and AI applicants may be able to obtain symbolic certification. Additionally, as with the low-risk program, the criteria must be carefully specified to ensure means-ends coupling.
 B. IEEE Ethics Certification Program for Autonomous and Intelligent Systems The IEEE ECPAIS program  [11]  is currently under development. It considers driving factors and inhibitors of AI system transparency, accountability, and bias. Offering a series of recommendations, the program is expected to yield a quality mark. The developer of an AI system can use the quality mark in presenting the system if it follows the recommendations in a checklist fashion. The program is expected to be administered by an organization other than the IEEE, which will assess certification applications. Whether the program will use an auditor to verify these applications has yet to be determined. The program is voluntary, with expected consumer demand as the driver for adoption. To be effective, EPCAIS would likely need the following. First, it would need to incentivize substantive adoption by generating consumer awareness and demand or by establishing an auditing requirement. Second, to improve means-ends coupling, it should offer distinct quality marks for transparency, accountability, and bias; it should also promote a systems mindset and consider the context in which the system is developed or deployed. Third, it should develop in-house expertise on certification; as far as the authors know, the IEEE has not launched a system-level certification program before. Finally, it should foster an enthusiastic audience in order to successfully launch. However, given that ECPAIS remains under development, we cannot offer a conclusive assessment.
 C. Malta AI Innovative Technology Arrangement The Maltese government launched the AI ITA certification program  [13]  in summer 2020. The program certifies AI systems to a particular use case following a review of an application detailing development and maintenance processes, corporate governance structures, and a responsible administrator. Applications must describe how the system handles failure modes and conforms to Malta's ethical AI framework, among other requirements  [13] . The program also (in most cases) requires applicant systems to have two software features: a "harness" that safeguards how the system handles specified failure modes and a "forensic node" that logs activity and is stored physically within Maltese jurisdiction. The applicant system must also disclose specific elements of its compliance with the ethical AI framework to end users via terms of service. Applications are reviewed by the Malta Digital Innovation Authority (MDIA) and are subsequently confirmed by an accredited auditor. The program is voluntary, with adoption expected to be driven by demand for certified systems from government procurers, businesses, and consumers. The Malta program has high feasibility because it benefits from a dedicated regulator which was established to manage this and similar programs. This dedicated expertise in assessing certification as well as accrediting third-party auditors can reduce concerns of symbol-substance decoupling. The program's focus on awarding certification for a specific system in a specific use case based on an assessment that includes the organization and a responsible individual incorporates a wide context that can help reduce concerns of means-ends decoupling. However, no application has yet been submitted. The effectiveness of the software harness to mitigate failure modes and the usefulness of mandated ethics disclosures in terms of service remain to be seen.
 D. Australian Chief Scientist Proposal: Turing Certification The Turing Certification program was proposed by Australia's Chief Scientist Alan Finkel in 2018  [12] ,  [71] . It would certify AI developer organizations and low-risk AI systems, drawing inspiration from FairTrade certification among other programs  [72] . This dual certification aspect could be especially impactful: it promotes a wide scope of assessment, thus improving means-ends coupling, and its organizational audit further improves symbol-substance coupling. As proposed, Turing Certification would be a voluntary program with third-party certifiers, with participation incentivized by demand from purchasers including consumers and governments. The third-party audit would confirm that the applying company, its development processes, and end AI system conform to not-yet-set standards for trustworthy AI. The proposal has not been enacted; it is unclear if it ever will be. Its most recent public discussion was in 2019  [73, p. 161 ]. If it is to be enacted, it would need clear certification criteria, a substantial challenge given the lack of agreed-upon trustworthy AI standards today.
 E. Queen's University Executive Education: Principles of AI Implementation In partnership with IEEE, Queen's University has launched an executive education certificate program on Principles of AI Implementation  [14] . Demand is likely to be driven by employers that value certified employees. The program has not yet run, and it is unclear how strong employer demand will be. The program covers ethical design principles, considerations for building trust in AI applications, and current Canadian regulatory requirements. This broad curriculum could support a systemic mindset, thereby improving means-ends coupling. However, these are dynamic topics; the program will need to stay up to date to retain means-ends coupling. Additionally, it is only a two-day program; this short duration precludes much substantive learning, creating the risk of symbol-substance decoupling.
 F. Finland Civics Course: Elements of AI The government of Finland, in a partnership between the University of Helsinki and the consultancy Reaktor, offers certificates to people who complete its civic education course Elements of AI. Certificates are awarded for all who complete at least 90% of the exercise and score 50% or above on attempted exercise questions  [74] . Credits can be transferred to the University of Helsinki. Demand may be driven by individuals' curiosity and sense of civic responsibility. It is unclear if employers would be interested in hiring certified individuals, and its introductory content would be insufficient for professional education, though that is not the program's aim. Thus far, demand has been significant: over 500 000 people have signed up and students from 170 countries have completed the course  [15] .
 G. Danish Labeling Program for IT Security and Responsible Use of Data The Confederation of Danish Industry, Danish Chamber of Commerce, SMEdenmark, and Danish Consumer Council  [16]  are currently developing a certification program for information technology security and data ethics. The program does not primarily focus on AI, but it was listed by the European Commission  [10, p. 10]  as an AI activity, and an interview and review of internal documents confirm that AI is within its scope. The program scope is ambitious, covering elements ranging from IT security to data management, to processes to address algorithmic bias. This broad scope could support a systemic mindset in the organization, and thus improve means-end coupling. The same scope may undermine symbol-substance coupling, however, as it could be difficult to achieve compliance with and effectively communicate to external stakeholders about so many certification criteria. The eventual administration of the program could encounter a challenge in moral hazard because a single organization-funded by certification fees-sets certification criteria, reviews applications, and performs (limited) audits.
 IV. CERTIFICATION FOR FUTURE AI TECHNOLOGY An essential attribute of AI technology is that it continues to evolve. Certification programs designed for the AI of today may not fare well with the AI of tomorrow. Adjustments will be needed to keep the programs up to date. However, adjustments can be expensive and are therefore not always made. Where possible, current certification programs should be designed to accommodate potential future directions in AI. This is no easy task: the future of AI cannot readily be predicted. Many have tried and failed  [75] . Nonetheless, some broad contours of future AI can be at least tentatively described, enough to derive some implications for certification programs. Toward the end of this section, we offer some remarks specific to the most advanced long-term AI, though much of the discussion may be more applicable to AI that will be developed in the interim, i.e., medium-term AI  [76] . Some elements of AI certification are relatively durable in the face of changes in the technology. First, many of the underlying goals for certification programs, such as fairness and beneficence, derive from universal ethics principles that apply not just to any AI technology, but to society in general. These ethics principles are used as a "North Star" guide in today's certification programs  [36, p. 35]  and will likely remain relevant as the technology matures. Future programs might potentially even certify the ethics of either the AI system or its developers. Second, some aspects of certification are matched to the types of people and institutions that are involved in AI, such as business executives, citizens, and corporations. It is fair to expect that such actors will remain involved in AI over the years. Certification program capacity to engage these actors may likewise remain relevant. That includes matters such as the capacity to improve corporate transparency and to educate large numbers of citizens. Similarly, one way for certification programs to remain relevant over time is to emphasize human and institutional factors. As discussed in Section II, certification programs can improve symbol-substance and means-ends coupling by promoting certain values and norms among applicants. Among other things, doing so positions applicants to act ethically as new challenges arise. Today, we do not know exactly what new challenges will arise for future AI, but we can be very confident that there will be new challenges. Certification programs should endeavor to make it more likely that whoever faces these new challenges will act according to a high ethical standard. Certification programs can also remain relevant by buildingin mechanisms for updating their certification criteria. For example, in the establishment of a new AI certification program, it could be specified that an expert body meets periodically to review and update the criteria, as is common with technical standards bodies. Programs could also include sunset provisions such that a program will cease to exist unless its criteria are updated. Additionally, certification programs could include a budget for ongoing research so that knowledge about how to update the criteria is available when needed. The field of AI research could also be a focus of certification. Future AI techniques will derive from current or future research. Some techniques may be better able to meet ethics standards; these techniques could be an object of certification. Academia already has institutional review boards to certify human subjects research as meeting standards for responsible conduct of research. Similar boards have been proposed for overseeing future AI research  [77] . Importantly, such boards should focus not just on whether the research meets procedural standards such as harm to research subjects, but also on whether it meets standards in terms of expected impacts on broader society  [78] . However, it can be difficult to know in advance which techniques will perform well-hence the need to do the research. Therefore, research is another domain in which certification may do well to emphasize human and institutional factors. Some forms of future AI may not be conducive to certification. In particular, AI systems that reach or exceed human-level intelligence in many domains  [79] ,  [80]  might not be a viable object of certification because humans could be powerless to take corrective actions suggested by the certification. To the extent that such AI is a significant concern, certification may only play a role in its development, while humans are still in command. This is one area in which certification of AI research programs may be especially significant, such as to promote trust among rival AI development groups  [81]  or share information about safety and ethics measures to alleviate rivals' concerns  [82] . Additionally, if the AI is developed by the national government-a distinct possibility given its national security implications-then there may be a role for an international certification regime, perhaps analogous to monitoring and verification regimes that have been established in other domains such as for biological, chemical, and nuclear materials. Should humanity ever face the prospect of developing such a technology, it would probably want to know that it was being developed according to a high ethical standard. That could be an important role for future AI certification programs.
 V. CONCLUSION Certification is a valuable tool for addressing information asymmetries and incentivizing better behavior. Certification can provide information about AI systems themselves as well as the organizations and individuals that are involved with them. AI-related programs could include both voluntary certification to ethics principles and mandatory conformity assessment to regulatory requirements. Today, a variety of AI certification programs are in development and use. These programs are focused on current AI technology, but certification can also play a role in future AI systems, up to including the most advanced long-term AI. One gap in the current AI certification landscape is the processes through which AI is developed. Although current certification programs assess the AI development process to varying degrees, no program certifies the process as its object. As a consequence, certification programs may fail to address process-oriented Potemkin attributes with maximal symbol-substance coupling and may fail to promote constructive organizational processes that can improve means-ends coupling. Improved attention to processes as the focus of certification could draw on, among other things, recent work on algorithmic auditing  [36]  and existing programs, such as ISO 9001 on quality management  [25]  and ISO 27001 on information security management  [83] . Early work at ISO on an AI management system standard may one day yield such a process-focused AI certification program  [84] . Another challenge is to improve the customization of AI certification for particular circumstances. Whereas some, such as the Malta AI ITA, are customized for specific jurisdictions, others, such as IEEE ECPAIS, make homogeneous prescriptions for AI developed worldwide. Additionally, most of the certification programs lack customization for each of the diverse sectors to which AI is applied, e.g., healthcare, transportation, and national security. Similarly, the programs tend to treat "AI" as a monolithic technology instead of being customized for the diverse types of AI that exist, e.g., language translation, facial recognition, and anomaly detection. Homogenous certification may similarly encounter challenges when an AI system is but a small, inextricable component of a larger digital system or workflow. All of this is a problem because certification programs that focus on specific niches tend to have better means-ends coupling  [21] . Many of the AI certification programs discussed here rely on, or at least strongly benefit from, demand from the customers of AI systems, including consumers, businesses, and governments. This is especially important for voluntary programs, where demand usually is the primary driver of participation. For good results to accrue, customers must not only demand certification-they must demand good certification. To that end, customers must be sufficiently educated about the technology and the associated ethics and safety concerns  [68, p. 23] . Education programs such as Finland's Elements of AI civics course could play an important role. Overall, AI certification remains at an early stage. Much remains to be seen, including which certification programs will be implemented, how effective they will be, how difficult or expensive they will be to implement, and how durable they will be in the face of changes in AI technology. All these aspects will influence the likelihood for adoption  [85]  and their success at symbol-substance and means-ends coupling. This article has presented a snapshot in time, with an assessment of an indicative sample of AI certification programs. It has also identified key concepts within the management literature on certification. Both contributions should inform future monitoring and evaluation of AI certification as the field matures. Future research would do well to track which certification programs are performing well, to learn from their successes and failures, and to identify gaps in the field and areas for improvement. It will likewise be important to monitor changes in AI technology and to ensure that certification programs stay up to date. AI certification is not a panacea, but it can play a valuable role in the overall mix of AI governance tools. TABLE II ASSESSMENT II OF CATEGORIES OF AI CERTIFICATION PROGRAMS details of the seven programs (see below) in the context of the management literature as reviewed above.
			 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 01:40:32 UTC from IEEE Xplore. Restrictions apply.
			 The authors thank Jonathan Aikman for this point.Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 01:40:32 UTC from IEEE Xplore. Restrictions apply.

Abstract: As artificial intelligence (AI) systems are increasingly deployed, principles for ethical AI are also proliferating. Certification offers a method to both incentivize the adoption of these principles and substantiate that they have been implemented in practice. This article draws from management literature on certification and reviews current AI certification programs and proposals. Successful programs rely on both emerging technical methods and specific design considerations. In order to avoid two common failures of certification, program designs should ensure that the symbol of the certification is substantially implemented in practice and that the program achieves its stated goals. The review indicates that the field currently focuses on self-certification and third-party certification of systems, individuals, and organizations-to the exclusion of process management certifications. Additionally, this article considers prospects for future AI certification programs. Ongoing changes in AI technology suggest that AI certification regimes should be designed to emphasize governance criteria of enduring value, such as ethics training for AI developers, and to adjust technical criteria as the technology changes. Overall, certification can play a valuable mix in the portfolio of AI governance tools.


Abstract: Open-universe probability models show merit in unifying efforts.


A PARAMETRIC, RESOURCE-BOUNDED GENERALIZATION OF L ÖB'S THEOREM, AND A ROBUST COOPERATION CRITERION FOR OPEN-SOURCE GAME THEORY

Pudlák shows that, for a consistent system, there exists some ε > 0 such that for all N , any proof of a statement of the form "there does not exist a proof of ⊥ using N or fewer characters" must use at least N ε characters. This means there is some obstruction to self-trust for a resource-bounded proof system, which suggests that a resource-bounded version of L öb's Theorem-a generic statement of selftrust about a family of sentences parametrized by a resource bound-might also hold. §2. Fundamentals. Since this article draws from work in several disciplines, this section is provided to clarify the use of notation and conventions throughout.
 Proof length conventions and notation. Proof length will be measured in characters instead of lines, the way one might measure the size of a text file on a computer. An extensive analysis of proof lengths measured in characters is covered by  [8] . Throughout this article, S refers to a fixed proof system (e.g., an extension of Peano Arithmetic). Writing S n φ, or simply n φ means that there exists an S-proof of φ using n or fewer characters.
 Proof system. Let S be any first-order proof system that 1) can represent computable functions (e.g., by being an extension of PA; see Section 2.4), 2) can write any number k ∈ N using O(lg(k)) symbols (e.g., using binary), and 3) allows the definition and use of abbreviations during proofs (see the Appendix for details). Compact numeral representations and abbreviations are allowed in the proof system for two reasons. The first is that real-world automated proof systems will tend to use these because of memory constraints. The second is that abbreviations make the lengths of shortest proofs slightly easier to analyze. For example, if a number N with a very large number of digits occurs in the shortest proof of a proposition, it will not occur multiple times; instead, it will occur only once, in the definition of an abbreviation for it. Then, one does not need to carefully count the number of times the numeral occurs in the proof to determine the contribution of its size to the proof length; the contribution will simply be linear in its length, or lg(N ). Write L S for the language of S, L S (r) for the set of formulas in L S with r free variables, and Const(S) for the set of closed-form constant expressions in S (e.g., 0, S0, S0 + S0, etc.). When φ ∈ L S (r), given any closed-form expressions c 1 , . . . , c r (such as constants, or free variables), write φ[ c] = φ[c 1 , . . . , c r ] for the result of substituting the c i for the free variables in φ. https://www.cambridge.org/core/terms. https://doi.org/10.1017/jsl.2017.42
 Choosing a Gödel encoding. Along with the proof system S, a single encoding #(−) : L S → N is chosen and fixed throughout, as well as a "numeral" mapping • (−) : N → Const(S) ⊆ L S for expressing naturals as constants in S. Note that in traditional PA, for example, • 5 = SSSSS0. However, to be more realistic it is assumed that S uses a binary encoding to be more efficient, so e.g., • 5 = 101. The maps #(−) and • (−) combine to form a G ödel encoding (−) : L S → Const(S) φ := • #φ which allows S to write proofs about itself.
 Representing computable functions. It is assumed that for any computable function f : N → N, there exists a "graph" formula Γ f ∈ L S  (2)  such that for all x ∈ N, S (∀y ) Γ f [ • x, y] ↔ y = • f(x) . For example, this condition holds if S is an extension of PA; see, e.g., Theorem 6.8 of Cori and Lascar (  [3] , Part II). Sometimes we abuse notation and write symbols for computable functions in-line with logical expressions to save space. For example, given functions f, g and h, to say that S proves that for any x value, f(x) < g(x) + h(x), technically one should write (∀x)(∀y1)(∀y1)(∀y3) Γ f [x, y 1 ] and Γ g [x, y 2 ] and Γ h [x, y 3 ] → y 1 < y 2 + y 3 but instead, we abuse notation and write (∀x) f(x) < g(x) + h(x)).
 Asymptotic notation. The notation f ≺ g will mean that for any M ∈ N, there exists an N ∈ N such that for all n > N, Mf(n) < g(n). The expression O(g) stands for the set of functions f g. If f : N → N is any specific function, f(O(g)) will stand for the set of functions of the form e • f where f ∈ O(g). §3. A bounded provability predicate, k . Here a predicate k is defined for asserting provability using a proof with length bounded by k.  (2)  exists that means, in natural language, that the number m encodes a proof in PA, and that the number n encodes the statement it proves. So, the standard provability operator : L PA → L PA can be defined as (φ) := (∃m)(Bew[m, φ ]).
 Defining k . Given a choice of G ödel encoding for Peano Arithmetic, it is classical that a formula Bew[m, n] ∈ L S It is taken for granted that Bew[m, n] exists for S and can be extended to a "bounded Bew" formula BBew[m, n, k] ∈ L S (3) that means • m encodes a proof in S, • n encodes the statement it proves, and • the proof encoded by m, before being encoded as m, uses at most k characters when written in the language of S. (Note that in general, m itself will be much larger than k as a member of N.) Then one can define a "bounded" box operator, which, given any sentence φ and closed expression k (such as a free variable, or a constant), returns k (φ) := (∃m)(BBew[m, φ , k]). Also taken for granted is a computable "single variable evaluation" function, Eval 1 : N → N, such that for any φ ∈ L S  (1) , Eval 1 ( φ , k) := φ( • k) . Since Eval 1 is computable, it can be represented in L S as in Section 2.4. This allows us to extend the k operator to act on sentences with one unbound variable. Specifically, if φ is a sentence with an unbound variable , then k (φ) := (∃m) BBew[m, Eval 1 ( φ , ), k]). Then k (φ) itself has as an unbound variable, and in natural language stands for "There is a proof using k or fewer characters of the formula φ".
 Basic properties of k . Each of the following properties will be needed multiple times during the proof of the main results. Since the proof is already highly symbolic, these properties are given English names to recall them.
 Property 1 (Implication distribution) . There is a constant c ∈ Const(S) such that for any p, q ∈ L S , (∀a)(∀b)( a (p → q) → ( b (p) → a+b+c (q))). Proof sketch. The fact that one can combine a proof of an implication with the proof of its antecedent to obtain a proof of its consequent can be proven in general, with quantified variables in place of the G ödel numbers of the particular statements involved. Let us suppose this general proof has length c 0 . Then, one needs only to instantiate the statements in it to p and q. However, if p and q are long expressions, it may be that they were abbreviated in the earlier proofs without lengthening them, so they can be written in abbreviated form again during this step. Hence, the total cost of combining the two proofs is around c = 2c 0 , which is constant with respect to p and q. Property 2 (Quantifier distribution). There is a constant C ∈ Const(S) such that for any φ ∈ L S  (1) , N ((∀k)(φ[k])) ⇒ (∀k) C +2N +lg(k) (φ[k]) , which in turn ⇒ (∀k) O(lg(k)) (φ[k]) . In the final subscript, O(lg(k)) stands for a closed expression representing a function of k that is asymptotically less than some positive constant times O(lg(k)). Proof. An encoded proof of φ[ • K] for a specific K can be obtained by specializing the conclusion of an N -character encoded proof of (∀k)(φ[k]) and appending the specialization with • K in place of k at the end. To avoid repeating • K numerous times in the final line (in case it is large), an abbreviation will be used for φ. Thus the appended lines can read as follows: (1) let Φ stand for φ , (2) Φ[ • K]. Let us analyze how many characters are needed to write such lines. First, a string Φ is needed to use as an abbreviation for φ. Since no string of length  N  2 has yet been used as an abbreviation in the earlier proof (otherwise one can shorten the proof by not defining and using the abbreviation), one can achieve Length(Φ) <  N  2 . As well, some constant c number of characters are needed to write out the system's equivalent of "let", "stand for", " ", and " ". Finally, lg(K) characters are needed to write • K. Altogether, the proof was extended by C + N + lg(k) characters, for a total length of 2N + c + lg(k). §4. A bounded generalization of Löb's Theorem. This section exhibits the first main theorem: a generalization of L öb's Theorem applicable to the analysis of resource-bounded proof systems. Property 4 (Bounded Inner Necessitation). For any φ ∈ L S , k (φ) → e(k) ( k (φ)). Then the following theorem holds:  (1)  be a formula with a single unquantified variable k, and suppose that f : Theorem 4.2 (Resource-bounded generalization of L öb's Theorem). Let p[k] ∈ L S N → N is computable and satisfies f(k) e(O(lg(k))). Then there is a threshold k ∈ N, depending on p[k], such that (∀k) f(k) (p[k]) → p[k] ⇒ ∀k > k (p[k]). Note: In fact a weaker statement, (∀k > k 1) f(k) (p[k]) → p[k] , is sufficient to derive the consequent, since we could just redefine f(k) to be 0 for k ≤ k 1 and then f(k) (p[k]) → p[k] is vacuously true and provable for k ≤ k 1 as well. The proof of Theorem 4.2 makes use of a version of G ödel's diagonal lemma that allows free variables to appear in the formula being diagonalized: Proposition 4.3. Suppose S is a first-order theory capable of representing all computable functions, as in Section 2.4. Then for any formula G ∈ L S (r + 1), there exists a formula ∈ L S (r) such that ∀ k [ k] ↔ G[ , k] , where k = (k 1 , . . . , k r ) are free variables. Proof. This result can be found on p. 53 of Boolos  [2] . Proof of Theorem 4.2. (In this proof, each centered equation will follow directly from the one above it unless otherwise noted.) We begin by choosing some function g(k) such that lg(k) ≺ g(k) and e(g(k)) ≺ f(k). For example, we could take g(k) = (lg(k))(e −1 (f(k))) . Define a formula G ∈ L S (2) by G[n, k] := (∃m) Bew[m, Eval 1 (n, k), g(k)]) → p[k] so that for any φ ∈ L S  (1) , G[ φ , k] = g(k) (φ[k]) → p[k]. Now, by Proposition 4.3, there is some ∈ L S  (1)  such that in some number of characters n, n (∀k)( [k] ↔ G[ , k]). ( 4.3) By Bounded Necessitation, n ((∀k)( [k] ↔ G[ , k])). By Quantifier Distribution, since n is constant with respect to k, (∀k) O(lg(k)) ( [k] ↔ G[ , k]) , in which we can specialize to the forward implication, (∀k) O(lg(k)) ( [k] → G[ , k]) . By Implication Distribution of O(lg(k)) , (∀k)(∀a) a ( [k]) → a+O(lg(k)) (G[ , k]) . By Implication Distribution again, this time of a+O(lg(k)) over the implication G[ , k] = g(k) (φ[k]) → p[k], we obtain (∀k)(∀a)(∀b) a ( [k]) → b g(k) ( [k]) → a+b+O(lg(k)) (p[k]) . Now we specialize this equation to a = g(k) and b = h(k), where h : N → N is a computable function satisfying e(g(k)) ≺ h(k) ≺ f(k), for example, h(k) = f(k)e(g(k)) : (∀k) g(k) [k] → h(k) g(k) ( [k]) → g(k)+h(k)+O(lg(k)) (p[k]) . https://www.cambridge.org/core/terms. https://doi.org/10.1017/jsl.2017.42 Then since g(k) + h(k) + O(lg(k)) < f(k) after some bound k > k 1 , we have (∀k > k 1) g(k) ( [k]) → h(k) g(k) ( [k]) → f(k) (p[k]) . Now, by hypothesis, (∀k) f(k) (p[k]) → p[k] , thus (∀k > k 1) g(k) ( [k]) → h(k) g(k) ( [k]) → p[k] . (4.4) Also, without any of the above, from Bounded Inner Necessitation we can write (∀k)(∀a) a ( [k]) → e(a) ( a ( [k])) . From this, with a = g(k), we have (∀k) g(k) ( [k]) → e(g(k)) g(k) ( [k]) . Now, since e(g(k)) < h(k) after some bound k > k 2 , we have (∀k > k 2) g(k) ( [k]) → h(k) g(k) ( [k]) . (4.5) Next, from Equations 4.4 and 4.5, assuming we chose k 2 ≥ k 1 for convenience, we have (∀k > k 2) g(k) ( [k]) → p[k] . (4.6) But from Equation  4 .3, the implication here is equivalent to [k], so we have N (∀k > k 2)( [k]), where N is the number of characters needed for the proof above. From this, by Bounded Necessitation, we have N ((∀k > k 2)( [k])). By Quantifier Distribution of N , (∀k > k 2) C +2N +lg(k) ( [k]) and since C + 2N + lg(k) < g(k) after some bound k > k, taking k ≥ k 2 for convenience, we have ∀k > k g(k) ( [k]) . (4.7) Finally, from Equations 4.6 and 4.7 we have, as needed, ∀k > k (p[k]). 4.1. Interpretation. L öb's Theorem may be viewed as an obstacle to a formal system of logic "trusting itself " to soundly prove any statement p. Previously, one might have thought this obstacle was merely a quirk of infinities arising from the unbounded proof-existence predicate . However, we see now that some bounded obstacle remains: namely, that a bounded logical system cannot trust itself "about moderately long proofs in general." To see this interpretation, let p[k] be any statement with a free parameter k, and f(k) e(O(lg(k))) be any function, representing "moderate largeness." Then the hypothesis (∀k) f(k) (p[k]) → p[k] of Theorem 4.2 says that our logical system generally trusts its proofs about p[k], even if they are moderately long. However, this will imply that ∀k > k, p[k], which is bad news if p[k] is sometimes false.
 Making f(k) small. The statement of Theorem 4.2 becomes stronger as one makes the function f(k) smaller, but it must remain e(O(lg(k))) for the theorem to apply. The obstruction to making f(k) small is hence the size of the proof expansion function e, which in real-world software for writing proofs about proofs will be under some design pressure to be made small, to manage computational resources. How small can e be made in practice? G ödel numberings for sequences of integers can be achieved in O(n) space (Tsai, Chang, and Chen,  [11] ) (where n is the length of a standard binary encoding of the sequence), as can G ödel numberings of term algebras  (Tarau, [9] ). To check that one line is an application of Modus Ponens from previous lines, if the proof encoding indexes the implication to which MP is applied, is a test for string equality that is linear in the length the of lines. Finally, to check that an abbreviation has been applied or expanded, if the proof encoding indexes where the abbreviation occurs, is also a linear time test for string equality. Thus, one can straightforwardly achieve e(k) ∈ O(k) for real-world theoremprovers. In that case, the condition f(k) e(O(lg(k))) amounts only to saying that f(k) lg(k). §5. Robust cooperation of bounded proof-based agents in the Prisoner's Dilemma. Bárász, Christiano, Fallenstein et al.  [1] , LaVictoire, Fallenstein, Yudkowsky et al.  [5] , and others have exhibited various agent-like logical formulae who can be viewed as playing the Prisoner's Dilemma by basing their "decisions" on proofs about each others' definitions (as strings). In particular, they proffer proof of the opponent's cooperation as an unexploitable condition for cooperation. However, their "agents" are purely mathematical entities who decide whether to cooperate based on undecidable logical conditions. This leaves open the question of whether their results are achievable by real software with bounded computational resources. So, consider the following program, where G : N → N is a function to be specified later: def FairBot k(Opponent) : let B = k + G(LengthOf(Opponent)) search for a proof of length at most B that Opponent(FairBot k) = Cooperate if found, return Cooperate else return Defect In this program, the subroutine "search for a proof of length at most B that (• • • )" is defined as a process which searches, in lexicographic order, through all strings of length ≤B, checking each string for whether it is a proof of ( up that is a proof of (• • • ), the search halts and sets found = true. If no such proof exists, the search continues until the (finite) set of strings of length ≤B have been exhausted, then halts and sets found = false. In what follows, all that matters is the functional behavior of the proof search procedure: that it sets found = true if a proof of length ≤B exists, and found = false otherwise. The program FairBot k may be viewed as a kind of proof-based "agent" that plays the Prisoner's Dilemma in the following sense. Given any string, Opponent, representing the source code of another program, we can compute the pair 1 R(FairBot k , Opp) := FairBot k (Opp), Opp(FairBot k )) . Viewed as such, FairBot k has the desirable property that the outcome R(FairBot k , Opp) = (Cooperate, Defect) will never occur: if its opponent defects, then FairBot will find no proof of its opponent's cooperation, so FairBot will itself defect. This property is called being unexploitable. At this point, a natural question arises: what happens when FairBot encounters a copy of itself ? That is, what is FairBot k (FairBot k )? Each FairBot will be searching for a proof that the other will cooperate. As such, one might expect to see a bottomless regression that will exhaust the proof bound B. ("The first FairBot must prove that the second FairBot must prove that the first FairBot must prove that. . . .") Thus, it seems like they will find no proof of cooperation, and hence defect. However, this turns out not to be the case. Letting p[k] := FairBot k (FairBot k ) = Cooperate) , it is a direct consequence of Theorem 4.2 that FairBot k (FairBot k ) = Cooperate. In fact a much stronger claim is true, as the next theorem will show. It will demonstrate a mutually cooperative program equilibrium (in the sense of Tennenholtz  [10] ) among a wide variety of (unequal) agents, provided only that they employ a certain principle of fairness, as follows.
 G-fairness. Given a nonnegative increasing function G, we say that an agent (i.e., program) A k taking a parameter k ∈ N is G-fair if for any opponent (i.e., program) Opp, we have k+G(LengthOf(Opp)) Opp(A k ) = C ) → A k (Opp) = C ) , where LengthOf(Opp) is the character length of the opponent's source code. In words, A k is G-fair if finding a short proof of its opponent's cooperation is a sufficient condition for A k to cooperate, when "short" is flexibly defined to be an increasing function of its opponent's complexity, i.e., k + G(LengthOf(Opp)). The agents FairBot k defined above are G-fair (where G is the function appearing in line 2 of their source code), and the reader is encouraged to keep them in mind as a motivating example for the following result:  n+a(m) (α[m, n]) → [n, m]. For later convenience, we also choose a nondecreasing computable function f(k) e(O(lg(k))) such that 6f(2 ) ≤ G( ). For example, we could take f(k) = G( lg(k) )/6 . Now, LengthOf(A k ) > lg(k) and LengthOf(B k ) > lg(k) since they reference the parameter k in their code. Applying G to both sides yields a(k), b(k) > G(lg(k)) ≥ 6f(k). (5.6) Define an "eventual cooperation" formula: p[k] := (∀m > k)(∀n > k)(α[m, n] and [n, m]). Using Quantifier Distribution once on the definition of p[k], (∀k) f(k) (p[k]) → (∀m > k)( C ((∀n > k)(α[m, n] and [n, m]))) where Adding these inequalities yields 3C + 4f(k) + 2lg(m) + lg(n) < n + a(m), so for some k 1 , from (5.7) we derive C = C + 2f(k) + lg(m). Applying Quantifier Distribution again, (∀k) f(k) p[k] → (∀m > k)(∀n > k)( C (α[m, n] and [n, m])) . ( 5 (∀k > k 1) f(k) (p[k]) → (∀m > k)(∀n > k) n+a(m) (α[m, n]) . Similarly, we also have 3C + 2lg(m) < m and 4f(k) + lg(n) < 5f(n) < b(n), so for some k 2 ≥ k 1 , (∀k > k 2 ) f(k) (p[k]) → (∀m > k)(∀n > k) n+a(m) (α[m, n]) and m+b(n) ( [n, m]) . (5.8) Thus by (5.5), (∀k > k 2) f(k) (p[k]) → (∀m > k)(∀n > k) c(n, m) and c(m, n)) , i.e., (∀k > k 2) f(k) (p[k]) → p[k] . Therefore, by Theorem 4.2 (and the note following it), for some k we have ∀k > k (p[k]). In other words, for all m, n > k + 1, A m (B n ) = B n (A m ) = Cooperate. Theorem 5.1 interesting for four reasons: 1. It is surprising. L öb's Theorem has not been applied much in the setting of game theory, and in fact at the time of writing, 100% of the dozens of mathematicians and computer scientists that the author has asked to guess the output of FairBot k (FairBot k ) have either guessed incorrectly (expecting the proof searches to enter an infinite regress and thus reach their bounds), or given an invalid argument for cooperation (such as "it would be better for the agents to cooperate, so they will"). 2. It is advantageous. When k is large, FairBot k outperforms the classical Nash/correlated equilibrium solution (Defect, Defect) to the Prisoner's Dilemma when facing itself (or any other G-fair agent) in a one-shot game with no iteration and no future reputation. 3. It is unexploitable. That is, the outcome (Cooperate, Defect) will never occur with a FairBot as player 1. If an opponent will defect against FairBot, FairBot will find no proof of the opponent's cooperation, so FairBot will also defect.
 It is robust. Previous examples of cooperative program equilibria studied by Tennenholtz  [10]  and Fortnow  [4]  all involved cooperation based on equality of programs, a very fragile condition. Such fragility is not desirable if we wish to build real-world cooperative systems. By contrast, the G-fairness criterion relies only on the provability of the opponent's cooperation, rather than details of its implementation, and therefore establishes mutual cooperation between a broad class of agents. §6. Conclusion. Theorem 4.2 represents a resource-bounded generalization of L öb's Theorem, which can be applied to algorithms that read and write proofs using bounded computational resources, such as formal verification software. Theorem 5.1 makes use of Theorem 4.2, and some additional proof-theoretic analysis, to demonstrate how algorithmic agents who have access to one another's source codes can inexploitably achieve cooperative outcomes that out-perform classical Nash equilibria and correlated equilibria. Moreover, the condition for cooperation in Theorem 5.1, called "G-fairness", is more robust than previously known unexploitable cooperative conditions, which depended on literal source-code equality  (Tennenholtz, [10] ). As a direction for potential future investigation, it seems likely that other agents described in the purely logical (noncomputable) setting of Bárász, Christiano, Fallenstein et al.  [1]  and LaVictoire, Fallenstein, Yudkowsky et al.  [5]  will likely have bounded, algorithmic analogs, and that many more general consequences of L öb's Theorem-perhaps all the theorems of G ödel-L öb provability logic-will have resource-bounded analogs as well. Definition 4 . 1 ( 41 Proof expansion function). Let e : N → N be any computable function bounding the expansion of S-proof lengths when they are G ödel encoded. That is, its definition is only that it must be large enough to satisfy the following two properties: Property 3 (Bounded Necessitation). For all φ ∈ L S ,
 Theorem 5 . 1 ( 5 . 2 . 1 . 51521 Robust cooperation criterion). Suppose that • e(k), the proof expansion function of our proof system as defined in Section 4, satisfies e(O(lg(k))) ≺ k, and • G( ) is any nondecreasing function satisfying G( ) e(O( )).Then, for any G-fair agents A k and B k , there exists some r 0 such that for all m, n > r, A m (B n ) = B n (A m ) = Cooperate. Feasibility of bounds in Theorem 5.Before the proof, recall from Section 4.2 that we can achieve e(k) ∈ O(k) for automatic proof systems that are designed for easy verifiability, in which case e(O(lg(k))) ≺ k, as needed.Proof of Theorem 5.1. The proof will make use of Theorem 4.2 at the very end, but first requires some additional proof-theoretic analysis. For brevity we leta(k) := G(LengthOf(A k )), (5.1) b(k) := G(LengthOf(B k )), (5.2) α[m, n] := A m (B n ) = Cooperate) , and (5.3) [n, m] := B n (A m ) = Cooperate) (5.4) so we can write the G-fairness conditions more compactly as m+b(n) ( [n, m]) → α[m, n] and (5.5)
			 Downloaded from https://www.cambridge.org/core. Carnegie Mellon University, on 24 Mar 2022 at 01:25:56, subject to the Cambridge Core terms of use, available at
			 https://www.cambridge.org/core/terms. https://doi.org/10.1017/jsl.2017.42 Downloaded from https://www.cambridge.org/core. Carnegie Mellon University, on 24 Mar 2022 at 01:25:56, subject to the Cambridge Core terms of use, available at
			 We must assume that Opp halts and returns either Cooperate or Defect in this case; note that FairBot k always halts. https://www.cambridge.org/core/terms. https://doi.org/10.1017/jsl.2017.42 Downloaded from https://www.cambridge.org/core. Carnegie Mellon University, on 24 Mar 2022 at 01:25:56, subject to the Cambridge Core terms of use, available at

Abstract:
This article presents two theorems: (1) a generalization of L öb's Theorem that applies to formal proof systems operating with bounded computational resources, such as formal verification software or theorem provers, and (2) a theorem on the robust cooperation of agents that employ proofs about one another's source code as unexploitable criteria for cooperation. The latter illustrates a capacity for outperforming classical Nash equilibria and correlated equilibria, attaining mutually cooperative program equilibrium in the Prisoner's Dilemma while remaining unexploitable, i.e., sometimes achieving the outcome (Cooperate, Cooperate), and never receiving the outcome (Cooperate, Defect) as player 1. §1. Introduction. In the game theoretic analysis of computerized agents who may read one another's source code, and more generally in the analysis of program verification systems that use proofs to verify other program verification systems, a need has arisen for a version of L öb's Theorem that applies to proof systems with bounded computational resources. The first main theorem of this article is a generalization of L öb's Theorem that applies in such cases, developed and proven in Section 2 through Section 4. The second main theorem, presented in Section 5, makes use of the first theorem and some further proof-theoretic analysis to derive some implications for the game theory of agents who read one another's source code. Specifically, game theoretic work of Bárász, Christiano, Fallenstein et al.  [1]  and LaVictoire, Fallenstein, Yudkowsky et al.  [5]  found that L öb's Theorem can be used to design entities in modal logic that resemble "agents" who achieve robust cooperative equilibria in games such as the Prisoner's Dilemma. However, these so-called "modal agents" were defined as strings representing uncomputable functions; strings of the form "if (• • • ) is provable, return 1, else return 0". It remained open whether their results would arise naturally for computable agents, a question answered affirmatively in Section 5. Pudlák [7]  on the lengths of proofs of finitistic consistency statements are suggestive of the first theorem of this article. Specifically,
 Related work. The work of