Siri Humphrey: Design Principles for an AI Policy Analyst

 Download as PDF

Abstract
This workgroup considered whether the policy analysis function in government could be replaced by an artificial intelligence policy analyst (AIPA) that responds directly to requests for information and decision support from political and administrative leaders. We describe the current model for policy analysis, identify the design criteria for an AIPA, and consider its limitations should it be adopted. A core limitation is the essential human interaction between a decision maker and an analyst/advisor, which extends the meaning and purpose of policy analysis beyond a simple synthesis or technical analysis view (each of which is nonetheless a complex task in its own right). Rather than propose a wholesale replacement of policy analysts with AIPA, we reframe the question focussing on the use of AI by human policy analysts for augmenting their current work, what we term intelligence-amplified policy analysis (IAPA). We conclude by considering how policy analysts, schools of public affairs, and institutions of government will need to adapt to the changing nature of policy analysis in an era of increasingly capable AI.
Introduction
Policy analysis is a common function in government in which public servants provide support for decision making with the aim of contributing to better decisions than would be made in the absence of such analysis. While decision makers have always relied on this support, the modern concept of the professional, multidisciplinary, policy analyst operating somewhere between the social scientist and the political operative was articulated by political scientist Harold Lasswell1 and has endured in much the same form since.
This workgroup was formed around the question can the policy analyst be replaced by artificial intelligence, and what are the implications for public policy and governance should this capacity be developed?2 As AI improves, can we imagine the displacement of human policy analysts by machines that understand questions asked by a decision maker, scour relevant databases for applicable knowledge, apply machine algorithms against diverse data to compute optimal strategies and make predictions about the impacts of various policy options, return an instant briefing, and engage in conversation about the issue with the decision maker?3
This note details the issues that emerged during our workgroup discussions. We start with a description of the current model for policy analysis and identify from there the design criteria for an AI replacement as the basis for identifying the specifications for such a system. Assuming that were developed and deployed, we consider how an artificial intelligence policy analyst (AIPA) would operate in the current model, leading to an assessment of its limitations. Rather than propose a wholesale replacement of policy analysts with AIPA, we focus on the use of AI by human policy analysts for augmentation and amplification of their skills—what we term intelligence-amplified policy analysis (IAPA). Thus, we revise our starting premise by considering how AIPA can serve as a supplement to the traditional human policy analysis competency. We conclude by considering how policy analysts, schools of public affairs, and institutions of government will need to adapt to the changing nature of policy analysis in an era of increasingly capable AI.
What is a Good Policy Analyst?
Policy analysis involves a range of activities including: identifying public concerns and determining their condition; assembling evidence, and analyzing potential public policy responses; projecting outcomes and developing strategies for dealing with trade-offs; constructing and evaluating options for addressing the problem; assembling bureaucratic and civil society coalitions to support policy formulation and implementation; communicating recommendations to decision makers; implementing the collective intention expressed in public policy decisions; and evaluating policies to measure effectiveness or value, and inform future responses to comparable policy problems.
During the first quarter century of the profession, logical positivism was dominant4 with a focus on the analysis of problems and potential interventions using techniques such as descriptive statistics, inference testing, modeling, operations research, systems analysis, and cost-benefit analysis becoming staples of the profession.5 Despite these advances, debates over the real, perceived, and proposed role of the policy analyst have coloured the profession's second quarter century. Stemming from some high profile failures of quantitative policy analysis to solve complex public policy problems, the post-positivist policy analysis movement called for a balancing of softer skills such as participatory design, stakeholder involvement, citizens' input, and qualitative methods alongside technical mastery.6 Graduate programs in public affairs continue to develop their students in a range of skills, but policy analysis remains a core competency for budding public servants7 — though whether or not this activity will continue to be valued in practice is less clear.8
There is a rich literature on what policy analysts should do, but less recent research on how policy analysts actually operate in practice.9 One survey of government policy analysts found that—when asked to rank five policy analyst archetypes (connector, entrepreneur, listener, synthesizer, technician) in order of how they understood and practiced their profession—the "synthesizer" archetype (defined in part as "consulting various sources to understand how a problem is conceptualized … develop recommended ways to deal with the problem") was most strongly identified with.10 We began our design considerations for an AIPA starting from the premise that much of the work of a policy analyst today involves this synthesis activity of responding to a question from a decision maker by finding information (which, for the practicing policy analyst today, usually starts with a Google search,11 complemented by social media listening12 and feeds from news aggregators) and synthesizing it in an easily-digestible form of decision support (e.g., a two-page briefing note). And while the "technician" archetype (defined in part as "locating of primary raw data sources in order to undertake statistical policy research") was ranked lowest in the above-noted survey, we also note the resurgence of policy analytics as a specialized input into evidence-based policy making13 and consider how an AIPA would support data-rich policy analytics in addition to information synthesis activity.
Policy analysts also engage in activities as a connector (i.e., developing cross-government and stakeholder support for policy solutions), listener (i.e., understanding how citizens and stakeholders feel about a specific policy issue), and policy entrepreneur (i.e., considering new conceptualizations of public problems, and developing creative and innovative solutions). Professional and experienced policy analysts develop domain expertise and situational awareness of a range of technical details in governing, the unwritten rules of how bureaucratic systems operate, and the political context that drives public policy concerns. Operationally, policy analysts maximize their principal's reach by engaging with, being responsive to, and channeling the politician's perspective. They protect their principal's interests by asserting the politician's perspective in internal and external fora, spotting potential threats, and addressing challenges before they become barriers. Policy analysts also interface with constituents, the broader public, stakeholders, interest groups, government colleagues, officials in other governments, and the politician's political colleagues, seeking to eliminate blindspots and highlight opportunities. Lastly, in an era of information abundance, policy analysts serve as a filter, guarding against unimportant information and directing the attention of decision makers towards important information.
Parts of the policy analyst's skill set (especially the synthesizer and technician elements) seem amenable to a machine approach to analysis and decision support. We focus on these policy analysis archetypes in assessing how an AIPA can be designed to replace the current model of the policy analyst. However, as we discussed these different views of what policy analysts do, we were reminded of what we knew already: other elements (i.e., connector, listener and policy entrepreneur)—and notably, the trust that develops between the policy analyst and the decision maker, which is necessary for an advisor's advice to have an impact on the decision maker's thoughts—are intensely human activities that do not seem likely to be overtaken by artificial intelligence in the foreseeable future.14 This foreshadows our conclusion, that a wholesale replacement of the policy analyst is not likely in the foreseeable technology environment. This does not, however, mean that AIPA that supports and supplements the work of human policy analysts is not achievable nor useful. However, we propose the idea of intelligence-amplified policy analysis (IAPA)15 as a near-term goal.
Design Criteria for an AIPA
Assuming the first task of the policy analyst is to synthesize information in response to a question posed by a decision maker, can AI be feasibly designed to replace this activity? We also considered the design of a technician-type AIPA, either as a direct machine interlocutor with a decision maker or as an assistant to a human policy analyst. What are the design criteria for an AIPA in performing these roles?
We start by sketching three use cases for such a system, asking how an AIPA16 might be asked to perform in a range of circumstances.
An Artificial Intelligence Policy Analyst: Use Cases in Three Orders of Government
Federal: 5G wireless and the hard decision of whether to ban Huawei
Our current standard for mobile networks—fourth generation, or 4G—will soon be overtaken by fifth generation, or 5G, wireless telecommunications. One issue has come to dominate the discussion around 5G: the Chinese telecommunications giant Huawei. The United States claims that Huawei is a threat to national security because of the as-yet-unproven claim that Huawei equipment contains a backdoor, and the data that moves through their equipment could be made available to Chinese intelligence services.
The Canadian federal government is currently considering Huawei's possible role as a vendor in the development of Canada's 5G systems. The government's decision is complicated by a number of factors: Canada's current, tenuous relationship with China; Canada's ongoing challenges with the United States on a number of trade issues; the Canadian tradition of internationalism, and of leading by keeping in step with others; domestic economic considerations; and uncertainty as to whether Huawei equipment poses a risk. If Huawei equipment does contain a backdoor, Huawei's central role in 5G could undermine the entire security of wireless telecommunications networks.
It is reasonable to assume that an AIPA could understand the questions being asked ("Hey Siri Humphrey, should Canada ban Huawei products in its developing 5G infrastructure?") and synthesize from a search of available materials a two-page briefing note on the relevant issues. This note is unlikely to be superior to what a human policy analyst could do, although it would likely be completed more quickly. Neither approach should conclude with an absolute recommendation, given the inherent uncertainty; rather, a small number of options should be identified, with the pros and cons of each. As with the human analyst, the decision maker could not be certain what biases may have influenced the content or emphasis in the briefing note.
As for technical policy analytics, we find little scope for an AIPA. As this decision is essentially binary – either ban Huawei equipment or allow it – there is little to nothing that can be experimented with before the decision. And even after the decision, the impacts are not of a big data variety but are rather significant event types (e.g., a major security breach from an exploited backdoor vulnerability; an escalation in Chinese displeasure should Huawei be banned, or American displeasure should it not be banned) that are better assessed through human analysis.
Provincial: Emergency room wait times
"Hey Siri Humphrey, how can we improve emergency wait times in the province?" asks the Health Minister. To answer this policy question, a wide range of data would be needed, including hospital administrative conditions (with variables such as shift structures, room capacity, equipment, support infrastructure, and expertise), community demographic data and public health conditions, and the availability of alternative health service delivery options such as health advice via telemedicine, community practitioners, and specialist physicians.
Automating the policy analysis in this case would require gathering all of this data, likely held in separate organizations, and creating a cost-benefit analysis for the province, citizens, stakeholders, doctors, and hospitals. The considerations to health of the people in the province, practical implications such as supply of skilled employees like doctors and nurses, and political considerations such as costs and labour concerns would all have to be weighed carefully. Research and answers to this policy question have been proposed by the medical community, but the political implications and decisions rest with the government of the day. Could Siri Humphrey find a way forward?
More likely, the tough political decisions related to cost, changing demographics of the population, and the decentralized implementation challenges have prevented greater policy action in reducing wait times. However, better application and use of data, gathered and synthesized by an AIPA could help inform advisors and politicians as they weigh careful options, and create better opportunities for small-scale policy experimentation to reduce wait times, without the political risk.
One approach to the question would focus on short term adaptation to demand for emergency room service, and the need to reallocate healthcare resources while still serving those in need of emergency care and those who don't have a community-based practitioner. Predictive analytics could help predict and plan for surges that may be imperceptible to hospital administrators, though this information only has value if hospital administrators have enough flexibility to respond to surges and lulls. AI could shift demand away from overburdened hospitals towards others with excess capacity by proactively steering patients based on an initial triaging of their condition and location (perhaps through a check-in app, although this would assume the Health Ministry can require patients to use the check-in app before being served in an emergency room).
But the question also implies a concern for longer-term investment and policy choices that might alleviate the root causes of the problem. Data sharing amongst hospitals, community practitioners, and the Ministry of Health could be used by the AIPA as the basis for policy recommendations, including the synthesis of information from other jurisdictions and settings.
Local: Public transportation options
Transit covers a range of subjects that can stretch the capacity of politicians and their staff, from routing and pricing to integrating mobility innovations into existing systems and tackling systemic inequality in transportation access. Substantive command of transit issues is only one part of managing the brief. At its best, transit policy development brings together technical concerns, public engagement, and maximizing not just mobility priorities but commercial, sustainability, equity, and other interests as well.
From an operational perspective, consider an alternative to how local bus transit services are run. Traditionally, transit planners analyze data (collected through surveys, rider counts, and demographic and traffic studies) and create routes and schedules and assign capacity to meet demand. Prices are set to balance fairness and willingness to pay with what it costs to run the system. Schedules and routes are then occasionally revised based on rider feedback, public consultation, and other data like rider counts. This approach is an example of traditional policy analysis. An alternative approach would be to offer on-demand transit services that use ML and algorithmic dynamic routing to respond to riders' requests for transportation—Uber for public transit, if you will. Such systems have already been deployed.17
For longer term planning purposes, automating policy analysis could give policy makers a more comprehensive view of the substantive landscape by integrating qualitative and quantitative data, historical performance and context, contemporary population and community trends, changes in demand for and gaps in service, expressions of public sentiment, and indicators of future demand. By having machines carry out time-intensive research and synthesis, policy staff could have more time to engage in public consultations, manage the complicated politics of transport, and develop more nuanced policy options for their principals. When combined with the output of other tools and technologies, staff and politicians could see how decisions will play out in real life in areas such as congestion, housing access and affordability, environmental impact, and job creation, letting them make subtle adjustments or strategic shifts to benefit favored groups or the general public interest.
As noted earlier, policy analysts currently use a standard web search as a part of their information gathering activities. Search is a relatively easy function to replicate in an AIPA. For more robust search functions that are not vulnerable to the business model of search-engine provider or gaming and manipulation of results by external actors (including data poisoning),18 recreating the policy analysis search function using something like the IBM Watson system19 would require populating an expert knowledge database that mirrors the information available to the human policy analyst.
While knowledge repositories have traditionally been difficult to populate and keep current,20 the Government of Canada's GCPedia system contains a repository of public servant knowledge21 that could serve as part of a database for training an automated search function. However, expert-populated databases face concerns over potential bias that machine algorithms can replicate and magnify. For example, IBM's celebrated Watson for Oncology system is sold as able to radically change cancer care, by analyzing massive amounts of diverse data (e.g., clinician notes, medical studies, clinical guidelines). Its treatment recommendations, however, are based on training by only about two dozen American clinicians whose hand-coded information on how patients with specific characteristics should be treated is used as the basis for Watson for Oncology recommendations.22
Social media listening can help to understand citizen's preferences, experiences, values, and behaviours in response to an actual or proposed policy tool change.23 Upon sending a signal into the policy environment (as either a proposed or real change), social media can be monitored to assess the reaction and adapt the signal in response, with citizen attitudes gauged and observed over time.24 Policy-relevant examples of this approach are steadily increasing.25 This approach to social media listening in support of policy analysis can be automated, built on the initial parameters that guide the machine approach (e.g., what terms to search for, how to evaluate sentiment, how to synthesize comments).26
Synthesis is a different challenge. While AI-driven narrative writing27 and document synthesis28 is improving rapidly, policy analysts' work involves deciding what information to filter out29 as well as what to summarize and how.30 Those filtering choices reflect the analyst's knowledge of the decision-maker's preferences, their professional judgments, and their biases. If used as training data, this knowledge, judgements, and biases would all become embedded in the AIPA's approach to synthesis, perpetuating and perhaps reinforcing the initial biases.
An AIPA could also conceivably act as a technician-type policy analyst, able to query a wide range of data sources, independently develop a model of system conditions and causal mechanisms, and predict future dynamics under either the status quo policy or a specified intervention. Current policy analytic systems scan environmental conditions and apply an algorithmically determined policy to achieve a specified objective.31 As the presence of sensors and devices in the social environment grows, capturing more data across a range of system conditions, opportunities for algorithmic approaches to monitoring and steering will also expand.32
Challenge: What constitutes relevant data and information?
Ironically, when policy analysts use Google to search for relevant information, they are already using AI. Google's RankBrain algorithm uses machine learning to process search results and provide relevant information to users, which contains its own inherent biases largely influenced by the business model underlying the search engine.33 The prospect of policy analysts relying on an AIPA to collect and sort information raises concerns about what information is of public relevance and what will become encoded in decision-making.
Embracing computational tools for policy work means subjecting human discourse, political valence, and knowledge to the procedural logic that undergirds all computation. As algorithms select what information is relevant to decision makers, they provide a knowledge logic34 — thereby gaining power over the flow of information and the assignment of meaning to it.
Policy analysts and builders of the system will need to think critically about what data, methods and models are for a given system. These choices are not neutral or objective, and will force policy advisors to make trade-offs. In order to make these trade-offs effectively, technical expertise will need to be strong not only for initial development but also for continuing adjustments after the system is deployed. The role of the policy analyst will need to evolve to meet this demand.
These shifting roles will pose challenges to two key tenets of democratic governments, transparency and accountability. How can you be accountable if you cannot explain why certain information was or was not considered for advice? A neural network such as BERT, developed by Google AI Language, is a state-of-the-art model for natural language processing (NLP) tasks like question answering, and might be the most accurate way to complete a given task, but may not be as explainable as a rule-based or decision-tree method. The appropriate trade-off between explainability and accuracy might depend on the nature of the advice being provided. Regardless of what method is chosen policy advisors should know what information is being given relevance over others.
Just as transparency is an important tenet of democracy, open code and methods are also valued in the software community. But if some citizens better understand the workings of the underlying method, greater transparency might enable them to game the system and effectively gain privileged access to government decision-makers, threatening increased inequality. The trade-offs between transparency and control are thus both important and context specific.
In deciding what data sources feed into an automated advisor system, policy analysts must consider the nature and origins of the data, in addition to judgments of relevance. These decisions cannot be fully neutral, but are always subject to power, politics, bias, and blind spots. How does social media distort what people actually care about?35 Who funds the think tank that produced a report? Security issues around data sources may may extend to the idea of data poisoning, where adversaries flood the system with misleading data or misinformation. It might be easier to recognize these concerns when analysts' time is freed up from collecting data and available to question and reflect on decision inputs. To improve their overall contextual awareness, policy advisors of the future will need to work in more multidisciplinary teams, including both data scientists and policy leaders.
Possibilities and Limits: Thoughts on a Feasible Current System
It is our judgment that it is feasible to build and use a synthesizer AIPA today, based on the following procedures and design elements:
1. The AIPA is given a strictly defined question to investigate based on current data; e.g., "What methods can be used to decrease homelessness in Edmonton?"
2. The AIPA performs a search on a series of textual data sources, including government reports, civil society advocacy and analysis reports, peer-reviewed academic journals, and miscellaneous materials sourced through Internet search engines.
3. The AIPA searches the data for text that may answer the given question. This is already feasible, using a BERT (Bidirectional Encoder Representations from Transformers)36 model trained on a sentence-pair question answering task, as is the case in the Stanford Question Answering Dataset.37
4. The text snippets containing relevant information are aggregated, together with the source data/articles, and presented to the policy analyst as a list, similar to the output of current legal e-discovery systems.38
5. The AIPA reviews this information, discards statements it judges irrelevant, and augments these statements with its own independent research. The scoring of each statement as useful or irrelevant is retained as feedback to the system to inform future tasks.
6. A GPT-2-style (Generative Pre-Trained Transformer version 2) generative text system39 trained on articles and past briefing notes is used to generate multiple draft briefing notes based on the discovered statements. It is important to highlight that this text synthesis is guided by the textual structure of briefing notes in the training data, not a human-level understanding of the question to be answered.
7. The human policy analyst selects one draft from amongst several as a starting point for the briefing note. Their decision patterns are recorded to inform further development of the system.
8. A human policy analyst then edits this note, adjusts it for factual accuracy, and makes subtle adjustments based on their emotional intelligence and understanding of the minister. The initial, GPT-2-generated draft and the final submitted note are both saved, to adjust the generative model using transfer learning so future notes more closely approach the style required by the Minister.
9. The Minister receives the briefing note, and provides feedback to further inform the process (as Ministers routinely do with the work of human policy analysts).
Assuming such a system is developed and deployed, how might an AIPA operate in our current political and administrative settings, and what might be its limitations?
These functions of search, synthesis, understanding, and document generation will become more advanced and reliable over time, due to both general technological advances and progressive specialization to the task at hand. As a result, the process will become increasingly streamlined, particularly for simple queries, so human analysts can focus on problems that require more subtle and complex synthesis of information – such as the question whether Canada should ban Huawei from 5G infrastructure. For the human analyst, this process will increasingly prioritize skills related to managing interpersonal relationships, strategic thinking, and understanding their audience.
Although the synthesizer is a dominant archetype in current views of policy analysis, creating briefing notes is a small part of what policy analysts do. Human policy advisors require skills that would be difficult to replicate with an AIPA, such as empathy and emotional intelligence. Human policy analysts must also be attuned to how the decision-maker thinks, in order to appreciate their personal perspective and anticipate their political concerns. They must balance the fragile dynamics of discretion and bias, seeking the equilibrium where the public good meets the public mood. This balancing act is an essentially human process in decision support, because even decisions that clearly appear to be in the public good sometimes require persuading citizens that this is so. Good policy advisors don't just provide the best options to decision makers, but also the frame in which they should be communicated. The best policy analysts also maximize their reach by engaging, being responsive to, and channeling the politician's perspective to constituents, interest groups, government officials, and political colleagues. They assert their perspective in internal and external fora, spotting and tackling threats and challenges, and projecting the vision of their principal.
There are additional limits to what an AI approach to policy advising and policy making can do. Making use of better information requires that ability and willingness to respond to it. Even with better information, sometimes governments lack the resources or flexibility to respond dynamically – like that hospital administrator receiving predictions of ER demand surges discussed above. Experimentation and design research is key to improving delivery, but not everything can be experimentally manipulated. A/B testing on websites is permissible, but performing randomized control trials using social welfare payments is much more problematic,40 even if the condition of social equipoise is met.41
The Intelligence-Amplified Policy Analyst (IAPA): Artificial Intelligence as Supplement (Rather Than Replacement) for the Policy Analyst
Given the limitations and challenges inherent in moving towards an IAPA, rather than propose the wholesale replacement of policy analysts with artificial intelligence, we argue that the foreseeable future of policy analysis will be one where human analysts us AI to augment and amplify their skills—an approach we term intelligence-amplified policy analysis (IAPA). We thus revise our starting premise, considering how AI can serve as a supplement to traditional policy analysis competencies. As such, we foresee an evolution, rather than a complete disruption of the strategic policy function in governments (barring near-term development of artificial general intelligence—AGI—and avoiding speculation about longer-term radical advances in machine intelligence). This evolution will require adaptation on the part of human policy analysts as there will be increased expectations on their abilities. With supplemental and assistive tools at their disposal, policy analysts may be expected to do more with fewer of their colleagues than currently.
Governments, however, should be cautioned that the IAPA is not a short-cut to cost savings: investments in systems and human capital will be required. For the education, training, and career development opportunities for public servants to provide a foundation and framework upon which to build a public service for the digital future, governments, universities, civil society, and public servants (both current and future employees) must work together to build new competencies, skills, and digital literacies (we conclude on this point, below).
Humans have been increasingly relying on external tools such as AI to achieve higher standards of productivity, and the benefits of this collaboration are being felt across a range of sectors. Yet being affected by AI does not mean that human beings will be replaced by technology, but that technology will be increasingly be used to assist people in their work. Therefore the need to upgrade human skills align with the requirements needed to work with AI will become essential. Howard Rheingold42 argues that human minds being replaced by a machine is not a phenomenon that the world will experience in the foreseeable future. Instead, he argues that the world will very soon experience the ubiquity of intelligence amplifiers, toolkits, and interactive electronic communities (today's social media) and that these tools will change how people think, learn, and communicate. Personal computers were an early manifestation of these amplifiers, opening up new avenues for people to manage the complexities of the modern world. Scaling up to broader usage, an economy can be said to be intelligently amplified when people are effectively trained in using ICTs to enhance their human intelligence. As the use of AI to augment human capabilities grows to sectors such as public administration, it is estimated that by the mid 2030s a third of public sector jobs will be affected by AI with senior officials and managers predicted to be most affected by automation.43
Intelligence amplification (IA) involves the adoption of artificial intelligence (AI) by knowledge workers as a complement to, rather than replacement of, their activities.44 Under an IA framework, human cognitive abilities are made more effective by adopting AI as a tool to support and amplify human intelligence. This view emphasizes the impact of advanced technology in expanding the capability of humans to understand complex problem situations and to derive robust solutions to problems more quickly.45 With the help of robotics and AI technologies, humans can be relieved of mundane work, leaving employees with more time to focus on areas that require intrinsically human skills such as creativity, ingenuity, and emotional intelligence.
The rapid growth in AI technology is gradually transforming into an essential component of work life today. Public sector organizations are incorporating AI capabilities to deliver services and increase effectiveness. The Government of Canada has already started using AI tools to accelerate human capabilities; its digital operations strategic plan envisions the need for government to manage digital tools and technology to augment human skills.46 A recent example can be found in the adoption of AI by Immigration, Refugees and Citizenship Canada to help with the department's backlog of immigration and refugee claims.47 The aim is for AI to eventually deal with refugee applications on its own and, if the project is successful, that front-line immigration officials can ultimately use this technology "to aid in their assessment of the merits of an application before decisions are finalized."48
Investing in the Future of AIPA and IAPA
We conclude by considering how policy analysts, schools of public affairs, and institutions of government will need to adapt to the changing nature of policy analysis in an era of increasingly capable AI. New skills and training for policy analysts will be required for the effective and knowledgeable use of AI in support of policy analysis. Skills in research design and investigating the data sources that inform decisions will increasingly be required. Additional skills include human-intensive work such as interviewing, engaging, and consulting to generate qualitative data sources. Human-intensive legwork, such as interviewing sources and engaging communities to gather qualitative data, will remain the domain of human policy analysts. However, the analysis of this data will be increasingly assisted by AI.
Policy analysts will be expected to merge traditional research skills with capacity in data analytics. In the near term, this will involve the crafting of new communication languages between specialist in data analytics and subject-matter expert policy analysts—a process known as paired analytics. As data analytics tools become more user friendly, generalist policy analysts will not only need to learn how to manipulate these tools but, more importantly, to know when they are using the tools incorrectly. Anyone who has ever used a statistical analysis computer package knows how easy it is to use a few simple commands to output reams of results that may or may not be meaningful.
Improvements in technical literacy skills among analysts will be required. Every analyst working with AI assistance must be able to understand the caveats and conditions that come with recommendations. Present-day AI education, given to current analysts and policy-makers and added to hiring criteria, will prepare the next generation of analysts for correctly interpreting AIPA processes and outputs. An important competency for policy analysts to develop is the ability to skeptically interpret AI recommendations and communicate their meaning to others. Human policy analysts will also be required to provide a source of accountability and ensure that recommendations minimize bias (subject to the acknowledgement that human policy analysts embody their own biases).
Perhaps surprisingly, the IAPA will have to improve their human interface skills as much as they will need to adapt their computer interface skills. As AI frees up their time, they will need to engage in forms of data collection that is more community and human centered. To illuminate those who may be digitally invisible49 to machine approaches, policy analysts will increasingly need to improve their street-level bureaucrat skills.
Governments will need to invest in and improve the data available to AIPAs. Data interoperability and sharing between departments and governments is a major limit to the development of AIPA. This might include making explicit the implicit knowledge that analysts use, or developing methods of quantifying "soft" data not traditionally available digitally and will allow AI to act in more culturally relevant ways.
Finally, successful movement towards IAPA will require acceptance of new processes by institutional and governmental leaders. While they will likely be removed from the day-to-day requirements of knowing how to correctly interpret the output of an AIPA, they must be willing to adapt to new processes for making and acting upon reports.
Next Steps
This note represents our collective response to the question posed at the outset—can the policy analyst be replaced by artificial intelligence, and what are the implications for public policy and governance should this capacity be developed?—largely written during our deliberations at the 2019 Summer Institute on AI and Society. Because of the fast-prototyping nature of the Summer Institute, this note does not perfectly articulate everything that was voiced during the working group's time together nor what could be accomplished with more time. As should be clear from the foregoing collection of many bullet points and threads of ideas, we have not yet developed a fully-coherent narrative to describe the many intersecting issues that our seemingly straightforward initial premise entailed.
Following from the Summer Institute and this note, we look forward to subsequent rounds of debate and writing that will build on the above, with the following specific outputs:
– The working group will produce a 1000 word think piece, for an outlet such as https://policyoptions.irpp.org, that speaks to the public policy practitioner community about any developing fantasies that AI is going to displace the policy analysis function, but also how the profession needs to adapt to changing possibilities. We are targeting a first draft by September 30, drafting collaboratively and video-meeting as necessary. A working title is "Relax, policy analysts—AI is not going to steal your jobs. But it is going to change how you work".
– An academic paper for a proposed Special Issue of the journal Canadian Public Administration.
– A creative piece (written as a script), tentatively titled "Yes, Siri Humphrey", will imagine a fully autonomous AIPA substituting for the character Sir Humphrey Appleby of Yes, Minister and Yes, Prime Minister fame.
– An academic grant proposal will be developed assembling a larger team to investigate the technical and applied aspects of continued developed of AIPA.The post Siri Humphrey: Design Principles for an AI Policy Analyst first appeared on AI Pulse.


Creating a Tool to Reproducibly Estimate the Ethical Impact of Artificial Intelligence

 Download as PDF

Abstract
How can an organization systematically and reproducibly measure the ethical impact of its AI-enabled platforms?1 Organizations that create applications enhanced by artificial intelligence and machine learning (AI/ML) are increasingly asked to review the ethical impact of their work. Governance and oversight organizations are increasingly asked to provide documentation to guide the conduct of ethical impact assessments. This document outlines a draft procedure for organizations to evaluate the ethical impacts of their work. We propose that ethical impact can be evaluated via a principles-based approach when the effects of platforms' probable uses are interrogated through informative questions, with answers scaled and weighted to produce a multi-layered score. We initially assess ethical impact as the summed score of a project's potential to protect human rights. However, we do not suggest that the ethical impact of platforms is assessed exclusively through preservation of human rights alone, a decidedly difficult concept to measure. Instead, we propose that ethical impact can be measured through a similar procedure assessing conformity with other important principles such as: protection of decisional autonomy, explainability, reduction of bias, assurances of algorithmic competence, or safety. In this initial draft paper, we demonstrate the application of our method for ethical impact assessment to the principles of human rights and bias.
Scope
The purpose of this document is to outline a method for assigning an ethical impact score to AI enabled platforms. One element of shared concern for corporations, and regulatory and soft-law organizations, is design of tools, including technical standards, for reproducible assessment of the ethical and social impact of AI projects. Presently, platforms with artificial intelligence ability are loosely governed by a patchwork of corporate policies and governmental regulations. They are also governed by a network of "soft law" requirements, such as standards issued by national standards bodies (NIST), international standards bodies such as the International Standards Organization (ISO), and by professional organizations such as the IEEE (the Institute of Electrical and Electronics Engineers). At present, both the ISO and IEEE are in the process of drafting or obtaining approvals for standards that govern the technical, ethical, and social impact of AI/ML platforms.
Standards are "documents that provide requirements, specifications, guidelines, or characteristics that can be used consistently to ensure that materials, products, processes, and services are fit for their purposes" (ISO). Standards provide stronger guidance than corporate policy or procedure statements. They are documents composed by volunteer experts working under normative principles such as consensus, non-domination, inclusion, and provisionalism.
Beyond offering a method to ensure compliance, standards can help organizations clarify processes that may otherwise be a "black box," which other stakeholders cannot replicate. Establishing methods that are transparent to multiple stakeholders is particularly important in fields like artificial intelligence or machine learning (AI/ML) – which raise deep social and ethical concerns that may implicate the economic and social sustainability of nations, organizations, or even humankind. In the case of AI/ML, where the technical nature of discussions can make them inaccessible to non-technical experts, having standards to help open the "black box" of related discussions, such as ethical impact discussions, is an avenue for much needed trust-building and transparency. While decisions about acceptable levels of risk of adverse impact can be forensically reconstructed from design teams' meeting notes, these reconstructions are limited by the detail of records and the quality of reporting tools. A well-characterized process that guides teams through discussions of the ethical implications of AI/ML, which may eventually be taken up as a standard, must go beyond this. The assessment tool we propose aims to guide these discussions, and to provide clear answers to the question, "what is the ethical impact of this AI-enabled platform?" – via a process that opens an otherwise inscrutable "black box."
Options for Assessing Ethical Impacts
What is "ethical impact"? This term is used in many, often vague, ways to describe negative effects of a technology on the lives of the people that use that technology. The ethical impact of a technology goes beyond its simple use, however, and should extend across the whole of the product's lifecycle and the lifespan of users. As understood here, ethical impact is the balance of positive and negative effects that a technology, whether in its developmental, design, deployment, or decommissioning stages, might have on the life choices and life chances of individuals as such or individuals in an aggregate like a company or school community.
There are at least two methods for assessing the ethical impact of AI-enabled platforms: a principles-based approach and a theories-based approach. A theories-based approach begins from the standpoint that ethical theories, like consequentialism or deontology, provide decision rules for making decisions under a specific vision of a good life. Used as guidelines for choices about platform impacts, ethical theories are most useful when the inputs, outputs, and effects are well-characterized. Ethical theories are not ideal, however, for making decisions under constraints of considerable uncertainty, wherein the pains and pleasures or roles and responsibilities cannot be clearly measured or integrated. Under uncertainty, a principles-based framework, under which a specific, well-defined principle is accepted axiomatically as an ideal to pursue, provides a more practical alternative approach. Principle-based frameworks avoid deep problems of ethical theory by moving comparisons of inter- or intra-personal utility off the table. It is thus possible to discuss the impact of a product or process in terms of its expected contribution to a specific dimension of a desirable state of affairs.
The assessment tool we have designed is intended to be a comprehensive approach to principle-based ethical impact assessment. It includes layers of questions with potential answers scored based on conformity with the relevant normative principle. The tool aims to elicit extensive consideration of a project's potential impacts, not just to provide a "check-box" task. Further, the tool is not intended to be a "one-off" or "single shot" evaluation, but rather to be revisited throughout the development cycle as new technical or human considerations emerge.
A "Human Rights First" Perspective
Initially, we adopt the perspective, already present in the well-known IEEE Ethically Aligned Design documents, that ethical AI projects must protect human rights foremost (IEEE Global Initiative 2017). This is not to deny the importance of other principles, but to elevate the importance of protecting human well-being as integral to the development and success of an AI-enabled future. With respect to human rights, we start from the perspective that the risks and benefits of an AI-enabled project can be evaluated using a set of questions derived from the 30 articles of the UN Declaration on Human Rights.
Arguments for the paramountcy of human rights abound, but there are few articulations of how to measure whether AI-enabled platforms adversely affect the life span, life chances, or life choices of rights holders. We reviewed the 30 components of the UN Declaration of Human Rights to determine whether each component raises specific ethical concerns of relevance to AI. As the thirty articles represent a panoply of legal and cultural issues that go beyond the scope of ethical assessment of AI, we sought to reduce the dimensions to a more manageable set. A team member with deep knowledge of the declaration proposed an aggregation of the 30 articles into five categories: general human rights, rights related to law and legality, rights related to personal liberty, rights related to political choice, and rights related to cultural and social choice. Our working arrangement of the articles into these five dimensions is shown in Box 1 below.
Box 1: 5 Dimensions of Rights and Associated Articles in the UN Declaration on Human Rights

To create a set of questions to probe the implications of an AI-enabled project for its potential to contravene any of these human-rights categories, we probed the conceptual schema of the first three articles – the general rights that represent pre-conditions for the remaining 27 rights – to identify distinct considerations within these groups of rights. This exercise generated seven broad guiding questions. For each of these, we created a set of more specific follow-up questions, which address concrete issues related to human rights protections. We list these questions in Box 2 below.
Box 2: Questions for Assessing the Human Rights Impact of AI-Enabled Projects

This "human rights first" approach brought into stark relief the challenges of crafting questions whose answers can be scored. This challenge arises most pointedly in the case of conceptual questions that admit a broader range of possible answers than a simple yes or no.
We then considered alternative principles, to assess the applicability of our method to principles with fewer dimensions, initially using the example of bias.
Alternative Principles Considered
Multiple organizations have issued statements of principles intended to govern artificial intelligence. Corporate entities, such as Accenture (Tan 2019), have put forth statements, as have governmental organizations. So too have multiple other organizations, chiefly professional associations in fields related to computer science and AI, such as ACM (Gotterbarn et al 2018) and IEEE.
The IEEE, under the remit of The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems (A/IS), has published their Ethically Aligned Design series of documents. The principles stated within this series of documents are:
– Human Rights: "A/IS shall be created and operated to respect, promote and protect internationally recognized human rights"
– Well Being: "A/IS creators shall adopt increased human well-being as a primary success criterion for development"
– Data Agency: "A/IS creators shall empower individuals with the ability to access and securely share their data, to maintain people's capacity to have control over their identity"
– Effectiveness: "A/IS creators and operators shall provide evidence of the effectiveness and fitness for purpose of A/IS"
– Transparency: "The basis of a particular A/IS decision should always be discoverable"
– Accountability: "A/IS shall be created and operated to provide an unambiguous rationale for all decisions made"
– Awareness of Misuse: "A/IS creators shall guard against all potential misuses and risks of A/IS in operation"
– Competence: "A/IS creators shall specify and operators shall adhere to the knowledge and skill required for safe and effective operation"
In addition, we considered the following principles, which are related to but not explicitly stated within the IEEE framework:
– Mitigation of bias
– Algorithmic competence
– Autonomy and consent for participants
– Safety
Multiple organizations within the ecosystem dedicated to ethical artificial intelligence and machine learning have proposed plans to translate these principles into practice. One example is the EU Governance Framework for Algorithmic Accountability and Transparency, which provides specific guidance to translate these two principles into regulatory governance of AI projects (European Parliamentary Research Service 2019). The EU Governance Framework does not, however, give organizations actionable measurements of these principles that would allow reconstructing principle-based decisions. Developing such a resource is the intended final outcome of this Ethical Impact Score project.
Method for Evaluating Ethical Impact
Whether an AI-enabled product or process will have a beneficial or adverse effect on its users will not be fully known until the product or process is used and its uses studied. Anticipating the possible effects on users' relationships to themselves or to other humans—the ethical impact—can be done through imaginatively questioning the developers about their expectations, then judging the answers to the questions given. Previous attempts to design an ethical impact measurement mechanism, such as the AI Ethics Toolkit (https://ethicstoolkit.ai/) have adopted the approach of asking questions about anticipated consequences of use.
These previously proposed mechanisms take two approaches: either they measure a project's overall level of ethical impact, or they measure a project's adherence with a single principle. Mechanisms in the first group, like the AI ethics toolkit, take the form of questionnaires with answers arrayed along an ordinal scale such as a Likert scale. The second group, like the EU Accountability framework, restricts responses to binary (yes/no) answers. It is our view that separating the scoring mechanism from the normative principles that motivate ethical concerns, for example focusing on auditability or risk, may lead to a "compliance" or "check-box" focused exercise. Providing sufficient specificity in relating questions to principles, and placing questions in the context of a sufficiently rich and reproducible, but numerically driven, scoring system, is a serious challenge that this draft only begins to address.
Scoring Mechanics
The Meaning of the Scores
Creating a numerical scoring mechanism for ethical impacts raises two types of concern: 1) that a numerical score may create a misleading sense of precision or confidence, and; 2) that a numerical score may be inappropriate for situations in which human wellbeing is at risk. With respect to the first concern, we stress that scores from the mechanisms proposed here should not be interpreted as establishing any unique threshold of acceptability: a project that receives a score of, say, 66 should not be regarded as more ethical than one with a score of 65. Instead, the Ethical Impact score shows development teams where there may be areas of concern. Through the use of our concept score, principle score, and final score, teams can identify where their projects may be falling short of a principle they aim to uphold. With respect to the second concern, AI-enabled platforms will have an undeniable effect on the lives (life span, life choices) of individuals and groups. The scores in this Ethical Impact Assessment mechanism are not meant to represent a path towards scoring or monetizing the value of those lives affected. The concept questions, particularly as they pertain to particular groups, are not intended as signals of those groups' value to others, including even to an AI system.
Question Design
The ethical implications of an AI enabled product or process cannot be fully captured through answers to one question per principle. Instead, we adopted a tiered approach to question development, to encourage teams to think through multiple layers of considerations, both technical and ethical. In the case of a human-rights first approach, the degree to which the product or process abides by or contravenes human rights is best captured, we propose, through questions that address each of the five rights categories we identified. These categories of rights lead to high-level questions, which are then augmented with questions associated with each of the concepts in the UN declaration articles and the interaction between those concepts. Similarly for other principles, questions to test the degree to which a product or process captures the principle are supplemented with substantive follow-up questions that aim to prompt users to consider the relationship between technical specifications and ethical considerations.
The design of the substantive sub-questions' response options invites a range of scoring options, including dichotomous (0 or 1) or other ordinal scales. The scores for sub-questions for a concept related to the overall principle will be combined to create a "raw concept score." "Raw concept scores" are based upon a formula for each concept based on the number of sub-questions and the relative importance of each sub-question to output a 0-5 score, then normalized to a score between 0-100 to ensure all questions have the same initial weight in the overall principle score. This raw score is transformed into a "weighted concept score" based on a within-concepts weighting scheme (see below). All weighted concept scores within a topic are then summed to yield an element within the final Impact Score.
The proposed scheme outputs three types of scores:
1. Concept scores: a summary score from 0 to 100 for each topic, based on responses to a set of concept questions and the relative importance attached to each question.
2. Principle scores: a final score from 0 to 100 based on the set of relevant concept scores, considering the relative importance of each concept to the team's beliefs about the principle.
3. Ethical Impact score: a final score from 0 to 100 based on the set of principle scores, taking into account the relative weight of each principle as determined by the project team
There are a number of advantages to this multi-level scoring scheme. First, the scheme allows a quick overall assessment of a given project or product. Second, by disaggregating a given overall score into scores related to specific principles, each of which can in turn be decomposed into responses to principle-specific questions, the scheme provides an expedient way to identify areas of concern that need improvement.
Weighting Scores
A key element of our Ethical Impact assessment tool is establishing a general scheme of weighting for a violation of each of the principles. The specific assignment of weights may vary, depending on the specific aim and deployment context of an AI system. There are two weighting schemes corresponding to the two types of scores that this tool will generate: Principle scores and an Ethical Impact score:
Within-principle weighting. The main idea is to tease out the concerns that animate a particular question, along dimensions of process and impact. The process dimension pertains to the processes of design, development and deployment of AI systems, with critical attention to potential divergence from industry standards and best ethical practices. The impact dimension pertains to potential adverse impacts of an AI system on the wider population, particularly on vulnerable groups. In each case, criticality is ranked from 1 to 5, with larger numbers denoting a stronger link to the principle in question. By averaging and normalizing across answers to principle-level questions one can assign a weight for a given principle for a project.
Variation in the number of questions across principles can reduce the effect of some questions on the Ethical Impact Score. To counteract this effect, some questions judged particularly important for a principle can be assigned high negative weights. For example, scoring low on a question like 1b below – did you establish a strategy or procedures to avoid creating or reinforcing unfair bias in the AI system, both regarding use of input data and algorithm design – will alert designers to rethink project elements so their system does not perpetuate bias. Low concept or principle scores, and a low overall Ethical Impact Score, should raise concerns to teams about the tenability of their project.
Between-principle weighting. Between-principle weighting will strongly affect the final Ethical Score for the project. Assigning weights to principles is likely to be more project-specific than assigning weights to questions within each principle. We contend that the organization or team developing a system should build internal consensus about these weights. This consensus can be built using various established methods (e.g., Delphi), to incorporate the views of external experts and avoid potential improper biasing of results.
Principle Assessment: Bias
As outlined above, we adopt a principle-based approach to evaluating the ethical impact of AI. Some principles, such as Accountability, have already been described by others in terms that are at least partially measurable. Others, such as protection of human rights and mitigation of bias, have not been. In this section, we propose a detailed set of questions, alternative answers, and scores for each answer, to create concept and principle scores for an AI system as it pertains to bias.
Bias
A major concern in AI ethics is bias: do systems produce different outcomes for identified groups, whether positive or negative. While considerations of bias are often stated as a unique concern, these are intertwined with principles of Human Rights, Well-Being, and Awareness of Misuse as described in the Ethically Aligned Design documents. In this section, we develop a tool to score evaluations of the considerations of bias.
We adopted a similar perspective to that we used for the "human rights first" perspective above, but here we add a potential numerical scoring system for answers to the substantive questions.

Reducing bias is only one component of a full evaluation of the ethical impact of AI enabled projects, of course. Other principles, and interactions among principles, must also be addressed.
Anticipated Future Work
Practical tools for ethical impact assessment are needed by multiple organizations, ranging from large professional bodies such as ACM and ICEE to small startups aiming to integrate ethical considerations with their technical work.
This draft tool is a starting point to fill this need. As presented here, the project is at approximately 40% completion and significant work needs to be done to accomplish all that is promised in this draft.
Crucial Near-Term Steps to take the project to 65-70% completion
– Develop a scoring and weighting scheme for a human-rights-first approach
○ The human rights first approach was introduced in this preliminary paper to illustrate how a complex, high-level principle can be broken into smaller, concept-focused, questions that might spur productive conversations about individual and community-level ethical impacts from AI-enabled projects. Identifying a range of possible answers and associated scoring mechanisms will require elaborating examples of human rights violations from other areas of society, including other technology-driven issues. Reasoning to a scoring and weighting mechanism from precedent seems an important component to appreciating the methods of argumentation in human rights law and ethics.
–  Develop guiding questions, component questions, and answer scoring and weighting schemes for additional principles.
○ Our work on additional principles is limited here by the short time available at the Summer Institute and the difficulty of organizing continuing work in a distributed environment (particularly where the project members struggled to fit this into their schedule at the start of a busy semester). The same issues impeded development of a weighting scheme for the first principle we considered, bias. Our near-term goals are to identify when more of the project team can work on a shared platform, such as a video conference, to address the weighting scheme and other principles.
Future Refinements Work to take this project to full completion
– Beta-testing usability of this tool in active project development
○ While the project team expects the usefulness of this tool to be high, we are not certain of the overall time burden or complexity of its use. Identifying a project team willing to work with us to test this tool is crucial to moving forward on areas of future refinements.
– Testing usability of this tool in an AI governance environment
○ The ultimate goal of this project is to move this Ethical Impact Assessment tool into the standards space. This would entail finding a working group sponsor, proposing the standard to that sponsor (including identifying a market for this standard), petitioning for the sponsor's support, establishing a working group, working with the sponsoring organization to develop the standard over a 2-3-year time frame, seeking approval of a developed standard, then drafting pathways for the dissemination and revision of this standard over time.
References
Dalkey, Norman; Helmer, Olaf (1963). "An Experimental Application of the Delphi Method to the use of experts". Management Science. 9 (3): 458–467. doi:10.1287/mnsc.9.3.458.
European Parliamentary Research Service (EPRS), Scientific Foresight Unit. (2019). "A Governance Framework for Algorithmic Accountability and Transparency". PE 624.262 – April 2019. Available at: http://www.europarl.europa.eu/RegData/etudes/STUD/<PHONE>/EPRS_STU(2019)624262_EN.pdf
Gotterbarn, D. et al. 2018. "Code of Ethics". Association for Computing Machinery. Available at: https://www.acm.org/code-of-ethics
IEEE Global Initiative for Ethics of Autonomous and Intelligent Systems (2017). Ethically Aligned Design of Autonomous and Intelligent Systems. Available at: ethicsinaction.ieee.org
Tan, C. 2019. "Putting AI Principles into Practice". Accenture Digital Perspectives. Available at: https://www.accenture.com/gb-en/blogs/blogs-organisations-start-ai-principles-practiseThe post Creating a Tool to Reproducibly Estimate the Ethical Impact of Artificial Intelligence first appeared on AI Pulse.


Shortcut or sleight of hand? Why the checklist approach in the EU Guidelines does not work

 Download as PDF

What we did
In April 2019, the High-Level Expert Group on Artificial Intelligence (AI) nominated by the EU Commission released the "Ethics Guidelines for Trustworthy Artificial Intelligence", followed in June 2019 by a second document, "Policy and investment recommendations". Initially our group was going to provide a formal response to the pilot phase of the April guidelines, however, as the pilot is geared more towards the business sector, our group undertook a critical assessment of the guidelines from a multidisciplinary perspective.
The Guidelines are aimed at supporting the development of 'trustworthy' AI, by: 1. outlining three characteristics (lawful, ethical, and robust); and, 2. providing seven key requirements (Human agency and oversight; Technical Robustness and safety; Privacy and data governance; Transparency; Diversity, non-discrimination and fairness; Societal and environmental well-being; and Accountability).
The Guidelines are a significant contribution to the growing international debate over the regulation of AI. Firstly, they aspire to set a universal standard of care for the development of AI in the future. Secondly, they have been developed within a group of experts nominated by a regulatory body, and therefore will shape the normative approach in the EU regulation of AI and in its interaction with foreign countries. As the General Data Protection Regulation has shown, the effect of this normative activity goes way past the European Union territory.
One of the most debated aspects of the Guidelines was the need to find an objective methodology to evaluate conformity with the key requirements. For this purpose, the Expert Group drafted an "assessment checklist" in the last part of the document: the list is supposed to be incorporated into existing practices, as a way for technology developers to consider relevant ethical issues and create more "trustworthy" AI.
In what follows we attempt to assess the implications and limitations of the assessment checklist for the global development of 'trustworthy' AI.
Prologue
During the 1969 Apollo mission to the moon, checklists played such a pivotal role in the logistical operations that astronaut Michael Collins referred to them as "The Fourth Crew Member." As the Apollo mission highlights, checklists are helpful for verifying the presence or absence of factors in many decision-making systems. Since checklists are effective tools for narrowing attention to essential information and eliminating distracting noise, the EU's "Ethical Guidelines for Trustworthy Artificial Intelligence," like many evaluative frameworks, relies on them. Unfortunately, such guidelines may fall victim to the fallacy of the checklist: by over-reductively framing complex problems, they can misleadingly suggest simple solutions to complex, indeed impossible, dilemmas. This poses the danger of creating a compliance regime that allows, even actively encourages, "ethics washing". Technology companies, and companies that use AI in their operations, can be incentivized to minimize their legal liability by touting their conformity to the inadequate guidelines, leaving the rest of society to pay the price for policy-makers not endorsing more nuanced tools.
The EU's Checklist Approach to AI Trustworthiness
The European Commission appointed a High-Level Expert Group on Artificial Intelligence, composed of representatives from academia, civil society, and industry, to support the implementation of the European Strategy on AI. The Expert Group's main remit was to propose recommendations for future policy development on ethical, legal, and societal issues related to AI. The outcome was the development of the Guidelines, with the Implementation List, or checklist, at their heart.
The Guidelines are supposed to contribute to a framework for achieving Trustworthy AI across application domains. The group identifies three components to trustworthy AI: it should be lawful, ethical and technically and socially 'robust'. The Guidelines focus on the ethical component, by first identifying the ethical principles and related values that must be respected in the development, deployment and use of AI systems. The Guidelines aim to operationalize the principles, by providing concrete (although non-exhaustive) guidance to developers and deployers on how to create more "Trustworthy" AI.
The justification of this overall structure stems from the idea that AI is a radically new technology – accurate perhaps in reference to its widespread application, if not to its intellectual lineage – which prompts new regulatory considerations. In this light, a step-by-step process starting with preliminary assessment by means of a checklist can reduce complexity, provide reassurance for developers and deployers, and so help develop trust.
The guidelines mention multiple caveats regarding the practicability, and – of particular importance – what is indicative and what is prescriptive in the structure of the assessment. What also becomes evident upon a closer examination, is that the Assessment List is a tool that may create a false sense of confidence, with little concrete guidance. Indeed, it may be seen as an outdated tool that might work in a cockpit, but in an unpredictable field like AI could end up failing.
Technical challenges to AI-related checklists
A key problem for the checklist is that it expects, or indeed requires, concrete answers to technical questions that are so nuanced, that any answer given is necessarily partial. The partial nature of the available answers leads to a certain amount of unavoidable uncertainty, which the checklist does not explain how to navigate. Yet, navigating this uncertainty is in fact the area where guidance is required in order to establish trustworthy AI. We elaborate on this dilemma with two examples: explanation and bias avoidance.
The guidelines ask practitioners to "ensure an explanation as to why the system took a certain choice resulting in a certain outcome that all users can understand." Currently, there is no way to definitively ensure that uncontestable explanations exist for a model. Generating explanations is an area of active research, and currently there is no consensus among experts as to what qualities make a good explanation, or even what an explanation consists of. Different methods of generating explanations provide different guarantees, they are often incomparable to each other (and can disagree) because they are based on different baseline assumptions. Thus, they are all partial explanations, with different strong and weak points. Further, the very notion of interpretability is often at odds with values such as security and privacy of information, which can lead to tensions in checking off boxes for both of these desired qualities.
Suppose an individual interacting with an AI system received an adverse outcome and wants an explanation for why the model exhibited this behavior. An example of this may be a resume scanner for job recruitment rejecting an individual from recruitment possibilities for a certain company.
One form of explanations are simply alternate inputs that have contrasting results – i.e. other resumes that were accepted by the given company's resume scanner. The individual can then intuit why the model made a given choice based on the differences they perceive in the two resumes. This type of explanation is highly intuitive and easy to understand, but it has its problems. One such problem is that there is no guarantee that any given difference perceived between the two resumes is in fact the model's reason for approving one and not the other. For example, it could be that the resume rejected was from a female candidate and the alternate input provided for explanation was from a male candidate. This may be the biggest difference perceived by the applicant, but it could be that the model is in fact not using this feature at all, but instead the change in hiring outcome was based on a tiny difference in work experience, if the model weights that very heavily. Thus, while explanations are intuitive to understand – it simply involves comparing two inputs – it does not guarantee that the correct reason for the difference in outcome between the two inputs is obvious from this comparison. Secondly, in the interest of privacy, which is another pillar of trustworthy AI according to the EU guidelines, one may want to not give a real alternate input for comparison but a fabricated one. Depending on how this alternate input is fabricated, it may be out of distribution of the hiring model's training set, making the model's response to this fabricated input inaccurate, making the entire explanation unreliable.
Another form of explanations are feature-based explanations, which seek to highlight which features (i.e. parts of an input – years of work experience, age, name on a resume) contributed the most to a model's output. In a linear model, this is straightforward—each feature has a weight associated with it and so the pathway from input feature to model decision is clear. However, releasing this information is a security risk that needs to be taken into account when using a linear model. Further, many applications, such as those involving images, require much more complex models. With these more complex models, feature-based explanations become more complicated, as even the question of what are the features becomes unclear, due to the fact that the model itself creates its own features as a means of understanding the input as an intermediate step in computation. Some feature-based explanations, in the interest of having more interpretable results, attribute importance to features that may not actually be used by the model. This is because the "features" investigated are defined some independent process (e.g. a person segmenting up an image) that is not necessarily the process that the model uses. [Simoyan, Selvaraju] Conversely, some more cautious forms of explanations potentially miss features that are used by the model.
Beyond this, depending on which baseline assumptions are used in determining which features are most important [Dhamdere, Kindermans, Sundararajan], different feature-based explanations may lead to different results, even if they agree on which features are to be used. It is unclear which are better or more reliable than another. Each method available currently has its own redeeming qualities, but also its own blind spots – and it is often confusing to use more than one type of explanation together because they can provide contrasting results.
Further, all types of explanations can be interpreted widely: there is no one way to interpret the common examples given or the influences of the features returned by a certain method. Even given the same explanation, researchers can come to contrasting conclusions about the behavior of a model. Thus, any explanation necessarily contains some uncertainty.
Navigating these nuanced tradeoffs–intuitiveness versus accuracy of an explanation, and privacy/security versus interpretability—are key places where guidance is required, but not supplied by the checklist. In order to make use of the methods currently available and navigate the inherent uncertainty of the answers they provide, we need to develop answers for questions such as: What purpose is the explanation supposed to serve? Is it for those interacting with the AI system? Is it so that they can understand what they can change about the input in order to get a different result? Is it so that they can be assured there was no mistake? Or unfair bias? Different explanation methods are suited to different applications. Once the purpose(s) are decided, what are the criteria for the explanation to be deemed satisfactory? At what point, and in what applications does a lack of explainability prohibit use of AI in a given context?
Another example of uncertainty comes in bias prevention, when the guidelines ask the developers to "ensure a working definition of 'fairness' that you apply in your AI systems". There are several definitions of what it means for an AI system to be free from bias, or fair [Hardt, Dwork]. While it is necessary to put these processes in place, they will have holes. Different auditing systems and notions of fairness, again, rely on different assumptions, often cannot be used together, [Kleinberg], and each have blind spots. Group-based fairness metrics, such as demographic parity and equalized odds, which seek to treat demographic groups similarly in aggregate (e.g., job recruitment rates should be equal across gender) sometimes sacrifice individual fairness, which seeks to treat similar people similarly on an individual level.
Additionally, these methods of checking for fairness do not often give obvious warning signs. Instead, algorithms often differ from each other in slight gradations, since many "baseline" or "fair" algorithms still have far from perfect scores on these fairness-checking tools. As such, it can be entirely unclear, even to an expert, when to flag an algorithm for exhibiting unfair behavior, outside of blatantly obvious cases.
Even if we use all compatible fairness notions together, there are still types of discrimination that we don't have good catches for—such as certain types of smaller subgroup discrimination. So no matter what kind of fairness definition a developer uses, whether they have caught discrimination in the model is uncertain. We need to have some way to deal with this built-in uncertainty, including defining the contexts where this uncertainty prohibits development.
Experts of the Future and Unacknowledged Uncertainty
The EU assessment list exists among other attempts to manage emergent technologies, such as foresight guidelines for nanotechnology. These are to be used by what Rose and Abi-Rached (2014) have called 'experts of the future', who 'imagine…possible futures in different ways, seeking to bring some aspects about and to avoid others. … In the face of such futures, authorities have the obligation not merely to "govern the present" but also to "govern the future" (p. 14). One way of governing the future is to build tools to manage the creation of new technologies, such as a checklist. Such tools, however, can be ill-suited for technologies where whether the box should be checked or not is uncertain, whether these uncertainties are caused by technological, social, or economic factors. The guidelines aim to provide shortcuts for managing complexity, but the shortcuts are inadequate given the affordances, uncertainty, and pace of AI development.
The Expert Panel acknowledges that the checklist is not a purely mechanistic enterprise or an exhaustive list. Yet in our view, the aspiration that the checklist integrates best practices in AI development and informs the debate unavoidably gives it a powerful orienting force, caveats about its "theoretical nature" notwithstanding. Alternatively, we could begin to think about the checklist in terms of its effect on people, behavior, and – even more importantly – judicial scrutiny of enterprises' activities in developing and commercializing emerging technologies. The checklist was a tool for the last industrial revolution, not the complexity, speed, and uncertainty of the next one.
Ethics on the Ground
The document overlooks the potential disjunction between idealistic ethical guidelines and their practical implementation – ethics on the ground. This echoes the gap between ambitious regulatory visions and the check-the-box corporate compliance that privacy scholars Deidre Mulligan and Kenneth Bamberger documented in their seminal book, Privacy on the Ground. It also elides the profound divide existing between ethics and trustworthiness.
It isn't clear what the consequences of these binary questions should be, other than noting that there could be tensions. The checklist as a technology was developed for handling complexity under stress. Experts provide pilots and astronauts checklists as memory aids. This is not a technology for deciding if you should build something at all or how to go about imagining what can be done ethically. Nor is this a technology for dealing with ambiguity. For example, what do you do if the answer to a question is maybe? How do you decide what to do if you can't guarantee that all users understand a decision? Do you then not develop the product at all? Is it likely that a team, having invested significant resources, will stop on a dime because they can't answer a question? More likely, they will simply pass it to the lawyers to finesse.
Thus, while the tool may add "ethical value" for the people who consider it, it does not impose sufficiently specific and consequential requirements to create trustworthiness. By virtue of its iterative path towards ethics assessment, the document orients itself toward developers and regulators, not the citizens and users whose lives AI will transform. With so much flexibility in interpreting and implementing the tool, and so much complexity and ambiguity in its target, it is natural that in practice, users and other stakeholders will gravitate to simpler versions of the tool. As a result, the checklist is likely to operate as a legalized template to legitimize questionable AI products and practices. A regulatory structure based on checklists will provide some protection, but protection for who? Users or an IT industry that has a practice of breaking things and people first, apologizing second, switching regulatory jurisdictions when they can, and paying fines if they have to? That is not a recipe for ethics by design or trustworthy AI. It is a recipe for "ethics-washing."
Overall, the report's guiding questions are valuable in identifying some of the most salient issues in AI ethics; they help identify issues and problems that all responsible actors in this space should attend to–issues that would be irresponsible to occlude or downplay. Nonetheless, this specific type of utility, heightened ethical awareness, is not what the Expert Panel claims for their framework. Rather, they characterize the framework as the foundation for creating and deploying trustworthy AI. This is more than semantics. Rather, conflating a trustworthy outcome with a more modest ethically attuned one risks encouraging stakeholders to develop misleading, overly-optimistic expectations which could ultimately lead to the opposite of trust–namely, betrayal.
References and Useful Links
Marvin Russell (2017), The History of Checklist: https://hackernoon.com/happy-national-checklist-day-learn-the-history-and-importance-of-october-30-1935-17d556650b89
Astronaut Checklists, https://cultureinfluences.com/en/process/english-guaranty-or-tool/english-astronaut-checklists/
Matthew Hersch, The Fourth Crewmember (2009), https://www.airspacemag.com/space/the-fourth-crewmember-37046329/
Atul Gwande, The Checklist, New Yorker, (2007) https://www.newyorker.com/magazine/2007/12/10/the-checklist
Daniel Greene, Anne Lauren Hoffman, and Luke Stark, Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning (2018): http://dmgreene.net/wp-content/uploads/2018/09/Greene-Hoffman-Stark-Better-Nicer-Clearer-Fairer-HICSS-Final-Submission.pdf
Thilo Hagendorff, The Ethics of AI Ethics (2019), https://arxiv.org/pdf/1903.03425.pdf
Deirdre Mulligan and Kenneth Bamberger, Privacy on the Ground (2015), https://mitpress.mit.edu/books/privacy-ground
Johns Hopkins University Center for Government Excellence Ethics and Algorithms Toolkit (Beta) (2019), https://ethicstoolkit.ai/ and https://drive.google.com/file/d/153f0TT_J4cDlr7LRTVKNIDZzu9JUa8ZI/view
Harvard Cyberlaw Clinic, Principled Artificial Intelligence Project (2019): https://clinic.cyber.harvard.edu/2019/06/07/introducing-the-principled-artificial-intelligence-project/
Paula Boddington, Towards a Code of Ethics for Artificial Intelligence: https://www.amazon.ca/dp/B077GCSKB1/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1
John Kleinberg, Inherent Trade-Offs in Algorithmic Fairness
https://www.researchgate.net/publication/330459391_Inherent_Trade-Offs_in_Algorithmic_Fairness
M. Hardt, Equality of Opportunity in Supervised Learning
https://ttic.uchicago.edu/~nati/Publications/HardtPriceSrebro2016.pdf
Cynthia Dwork, Fairness Through Awareness
https://arxiv.org/abs/<PHONE>
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron? CoRR, https://arxiv.org/abs/1805.12233
P.-J. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim. The (Un)reliability of saliency methods. ArXiv e-prints, November 2017
Rose, N., & Abi-Rached, J. M. (2013). Neuro: The new brain sciences and the management of the mind. Princeton: Princeton University Press.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ArXiv e-prints, 2017.
Simonyan et al. "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps"
Selvaraju et al. "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization"The post Shortcut or sleight of hand? Why the checklist approach in the EU Guidelines does not work first appeared on AI Pulse.