This seems very related to the notion of an "enforced standard" in the previous paper, though here it is only necessary to enforce a _goal_ across everyone, and the details of processes can vary across regulators. I especially like the scenario in which regulators emerge "bottom-up" from researchers thinking about potential problems with AI, though I'm not sure how likely it is.

With both this and the previous paper, I can see how they would apply to e.g. self-driving cars and adversarial robustness, but it's less clear to me how such an approach can help with AI alignment. If we believe that alignment is really hard and we only get one shot at it, then it seems especially difficult to have legible regulations that ensure, **without any testing**, that we don't build a misaligned superintelligent AI. Alternatively, if we believe that we will have lots of non-catastrophic experience with aligning AI systems, and can iterate on our processes, then it seems more likely that we could develop useful, legible regulations. (I am more inclined to believe this latter scenario, based on <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) and other intuitions.) Even in this scenario I don't yet know what regulations I would place, but it seems likely that with more experience we would be able to develop such regulations.

Regulatory Markets for AI Safety

This paper presents an idea on how AI could be regulated: by the introduction of a **market of private regulators** that themselves are regulated by the government. Companies would be required by law to purchase regulatory services, but could choose which regulator they purchase from. The regulators compete to attract companies, but are all required to meet goals set by the government.

The key benefit of such an approach is that the government now only needs to set **goals** for regulation (e.g. for self-driving cars, a limit on the rate of accidents) while offloading to private regulators the regulations on **processes** (e.g. required adversarial training on the vision models employed in self-driving cars). This relieves the burden on government, which is currently too slow-moving to effectively regulate AI. It gets the best of both worlds: as with government regulation, it can optimize for the public good, and as with tech self-regulation, it can have best practices emerge from the researchers who know best (since they can build their own regulatory startups).

Of course, for this to work, it is crucial that the private regulators avoid regulatory capture, and that **the market for regulators is competitive and independent**.


It does seem like building trust between actors is a pretty key part of AI policy. That said, there are two kinds of trust that you can have: first, trust that the statements made by other actors are true, and second, trust that other actors are aligned enough with you in their goals that their success is also your success. The former can be improved by mechanisms lie monitoring, software verification, etc. while the latter cannot. The former is often maintained using processes that impose a lot of overhead, while the latter usually does not require much overhead once established. The former can scale to large groups comprising thousands or millions of people, while the latter is much harder to scale. I think it's an open question in AI policy to what extent we need each of these kinds of trust to exist between actors. This podcast seems to focus particularly on the latter kind.

Other miscellaneous thoughts:
    - I think a lot of these views are conditioned on a gradual view of AI development, where there isn't a discontinuous jump in capabilities, and there are many different actors all deploying powerful AI systems.
    - Conditional on the military eventually developing AI systems, it seems worth it to work with them to make their AI systems safer. However, it's not inconceivable that AI researchers could globally coordinate to prevent military AI applications. This wouldn't prevent it from happening eventually, but could drastically slow it down, and let defense scale faster than offense. In that case, working with the military can also be seen as a defection in a giant coordination game with other AI researchers.
    - One of my favorite lines: "I would recommend everyone who has calibrated intuitions about AI timelines spend some time doing stuff with real robots and it will probably … how should I put this? … further calibrate your intuitions in quite a humbling way." (Not that I've worked with real robots, but many of my peers have.)

80K podcast: How can policy keep up with AI advances?

Rob Wiblin, Jack Clark, Miles Brundage and Amanda Askell

OpenAI policy researchers Jack Clark, Amanda Askell and Miles Brundage cover a large variety of topics relevant to AI policy, giving an outside-view perspective on the field as a whole. A year or two ago, the consensus was that the field required [disentanglement research](https://forum.effectivealtruism.org/posts/RCvetzfDnBNFX7pLH/personal-thoughts-on-careers-in-ai-policy-and-strategy); now, while disentanglement research is still needed, there are more clearly defined important questions that can be tackled independently. People are now also taking action in addition to doing research, mainly by accurately conveying relevant concepts to policymakers. A common thread across policy is the framing of the problem as a large coordination problem, for which an important ingredient of the solution is to build _trust_ between actors.

Another thread was the high uncertainty over specific details of scenarios in the future, but the emergence of some structural properties that allow us to make progress anyway. This implies that the goal of AI policy should be aiming for _robustness_ rather than _optimality_. Some examples:
    - The [malicious use of AI report](https://maliciousaireport.com/) was broad and high level because each individual example is different and the correct solution depends on the details; a general rule will not work. In fact, Miles thinks that they probably overemphasized how much they could learn from other fields in that report, since the different context means that you quickly hit diminishing returns on what you can learn.
    - None of them were willing to predict specific capabilities over more than a 3-year period, especially due to the steep growth rate of compute, which means that things will change rapidly. Nonetheless, there are structural properties that we can be confident will be important: for example, a trained AI system will be easy to scale via copying (which you can't do with humans).
    - OpenAI's strategy is to unify the fields of capabilities, safety and policy, since ultimately these are all facets of the overarching goal of developing beneficial AI. They aim to either be the main actor developing beneficial AGI, or to help the main actor, in order to be robust to many different scenarios.
    - Due to uncertainty, OpenAI tries to have policy institutions that make sense over many different time horizons. They are building towards a world with formal processes for coordinating between different AI labs, but use informal relationships and networking for now.

AI policy is often considered a field where it is easy to cause harm. They identify two (of many) ways this could happen: first, you could cause other actors to start racing (which you may not even realize, if it manifests as a substantial increase in some classified budget), and second, you could build coordination mechanisms that aren't the ones people want and that work fine for small problems but break once they are put under a lot of stress. Another common one people think about is information hazards. While they consider info hazards all the time, they also think that (within the AI safety community) these worries are overblown. Typically people overestimate how important or controversial their opinion is. Another common reason for not publishing is not being sure whether the work meets high intellectual standards, but in this case the conversation will be dominated by people with lower standards.

Miscellaneous other stuff:
    - Many aspects of races can make them much more collaborative, and it is not clear that AI corresponds to an adversarial race. In particular, large shared benefits make races much more collaborative.
    - Another common framing is to treat the military as an adversary, and try to prevent them from gaining access to AI. Jack thinks this is mistaken, since then the military will probably end up developing AI systems anyway, and you wouldn't have been able to help them make it safe.
    - There's also a lot of content at the end about career trajectories and working at OpenAI or the US government, which I won't get into here.


This post makes an important point in a clear and concise way. My only concern is that "structural problems" is such a broad heading that practically anything can be included, making it more difficult to specifically direct attention towards existential threats (the same is true for the term "accidents", which to me doesn't properly reflect the threat of adversarial behaviour from AI). I don't know how to best handle this tradeoff, but think it's a point worth raising.

*Rohin's opinion*: I just wanted to add a note on why we've highlighted this piece. While many of the particular concrete examples have been explained before, the underlying system for thinking about AI is new and useful. I particularly liked the distinction made between focusing on _agency_ in AI (which leads you to think about accidents and misuse) vs. thinking about incentives and structure (which leads you to think about the entire causal chain leading up to the moment where an agent causes something bad to happen).

Thinking About Risks From AI: Accidents, Misuse and Structure

The authors argue that in addition to risk from misuse of AI and "accidents", we should pay attention to the structural perspective: how AI changes the broader environment and incentives of various actors. Possible examples include creating winner-take-all competition or creating overlap between offensive and defensive actions. In the face of these effects, even competent and well-intentioned decision-makers might be pressured into making risky choices. To ameliorate this problem, more people should focus on AI policy, particularly social scientists and historians; and we should think hard about creating collective norms and institutions for AI.


AI Governance: A Research Agenda

A comprehensive document about the research agenda at the Governance of AI Program. This is really long and covers a lot of ground so I'm not going to summarize it, but I highly recommend it, even if you intend to work primarily on technical work.


I'm very excited by this charter, it's a good sign suggesting that we can get the important actors to cooperate in building aligned AI, and in particular to avoid a competitive race. Key quote: "if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project".

OpenAI Charter

In their words, this is "a charter that describes the principles we use to execute on OpenAI's mission".


One technique I like a lot is backchaining: starting from the goal you are trying to accomplish, and figuring out what actions or intermediate subgoals would most help accomplish that goal. I’ve spent a lot of time doing this sort of thing with AI alignment. This paper feels like it is advocating the same for AI governance, but also gives a bunch of concrete examples of what this sort of work might look like. I’m hoping that it inspires a lot more governance work of the problem-finding variety; this does seem quite neglected to me right now.

One important caveat to all of this is that I am not a governance researcher and don’t have experience actually trying to do such research, so it’s not unlikely that even though I think this sounds like good meta-research advice, it is actually missing the mark in a way I failed to see.

While I do recommend reading through the paper, I should warn you that it is rather dense and filled with jargon, at least from my perspective as an outsider.

‘Solving for X?’ Towards a problem-finding framework to ground long-term governance strategies for artificial intelligence

The typical workflow in governance research might go something like this: first, choose an existing problem to work on; second, list out possible governance mechanisms that could be applied to the problem; third, figure out which of these is best. We might call this the _problem-solving_ approach. However, such an approach has several downsides:

1. Such an approach will tend to use existing analogies and metaphors used for that problem, even when they are no longer appropriate.
2. If there are problems which aren't obvious given current frameworks for governance, this approach won't address them.
3. Usually, solutions under this approach build on earlier, allegedly similar problems and their solutions, leading to path-dependencies in what kind of solutions are being sought. This makes it harder to identify and/or pursue new classes of solutions
4. It is hard to differentiate between problems that are symptoms vs. problems that are root causes in such a framework, since not much thought is put into comparisons across problems
5. Framing our job as solving an existing set of problems lulls us into a false sense of security, as it makes us think we understand the situation better than we actually do ("if only we solved these problems, we'd be done; nothing else would come up").

The core claim of this paper is that we should also invest in a _problem-finding_ approach, in which we do not assume that we even know what the problem is, and are trying to figure it out in advance before it arises. This distinction between problem-solving and problem-finding is analogous to the distinction between normal science and paradigm-changing science, between exploitation and exploration, and between "addressing problems" and "pursuing mysteries". Including a problem-finding approach in our portfolio of research techniques helps mitigate the five disadvantages listed above. One particularly nice advantage is that it can help avoid the [Collingridge dilemma](https://en.wikipedia.org/wiki/Collingridge_dilemma): by searching for problems in advance, we can control them before they get entrenched in society (when they would be harder to control).

The authors then propose a classification of governance research, where levels 0 and 1 correspond to problem-solving and levels 2 and 3 correspond to problem-finding:

- **Business as usual** (level 0): There is no need to change the existing governance structures; they will naturally handle any problems that arise.
- **Puzzle-solving** (level 1): Aims to solve the problem at hand (something like deepfakes), possibly by changing the existing governance structures.
- **Disruptor-finding** (level 2): Searches for properties of AI systems that would be hard to accommodate with the existing governance tools, so that we can prepare in advance.
- **Charting macrostrategic trajectories** (level 3): Looks for crucial considerations about how AI could affect the trajectory of the world.

These are not just meant to apply to AGI. For example, autonomous weapons may make it easier to predict and preempt conflict, in which case rather than very visible drone strikes we may instead have "invisible" high-tech wars. This may lessen the reputational penalties of war, and so we may need to increase scrutiny of, and accountability for, this sort of "hidden violence". This is a central example of a level 2 consideration.

The authors note that we could extend the framework even further to cases where governance research fails: at level -1, governance stays fixed and unchanging in its current form, either because reality is itself not changing, or because the governance got locked in for some reason. Conversely, at level 4, we are unable to respond to governance challenges, either because we cannot see the problems at all, or because we cannot comprehend them, or because we cannot control them despite understanding them.


See the next summary. Note also that the author is organizing the <@Cooperative AI Workshop@> to tackle some of these issues.

AI Governance: Opportunity and Theory of Impact

What is the theory of change for work on AI governance? Since the world is going to be vastly complicated by the broad deployment of AI systems in a wide variety of contexts, several _structural risks_ will arise. AI governance research can produce "assets" (e.g. policy expertise, strategic insights, important networking connections, etc) that help humanity make better decisions around these risks. Let's go into more detail.

A common perspective about powerful AI is the "superintelligence" perspective, in which we assume there is a single very cognitively powerful AI agent. This leads people to primarily consider "accident" and "misuse" risks, in which either the AI agent itself "wants" to harm us, or some bad actor uses the AI agent to harm us.

However, it seems likely that **we should think of an ecology of AI agents, or AI as a general purpose technology (GPT)**, as in e.g. <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) or [Age of Em](https://ageofem.com/). In this case, we can examine the ways in which narrow AI could transform social, military, economic, and political systems, and the _structural risks_ that may arise from that. Concrete examples of potential existential structural risks induced by AI include nuclear instability, geopolitical turbulence, authoritarianism, and value erosion through competition.

A key point about the examples above is that the relevant factors for each are different. For example, for nuclear instability, it is important to understand nuclear deterrence, first strike vulnerability and how it could change with AI processing of satellite imagery, undersea sensors, cyber surveillance and weapons, etc. In contrast, for authoritarianism, relevant processes include global winner-take-all-markets, technological displacement of labor, and authoritarian surveillance and control.

This illustrates a general principle: unlike in the superintelligence perspective, **the scope of both risks and solutions in the ecology / GPT perspectives is very broad**. As a result, we need a broad range of expertise and lots of connections with existing fields of research. In particular, **"we want to build a metropolis -- a hub with dense connections to the broader communities of computer science, social science, and policymaking -- rather than an isolated island"**.

Another important aspect here is that in order to _cause better decisions to be made_, we need to focus not just on generating the right ideas, but also on ensuring the right ideas are in the right places at the right time (e.g. by ensuring that people with the right tacit knowledge are part of the decision-making process). Instead of the "product model" of research that focuses on generating good ideas, we might instead want a "field-building model", which also places emphasis on improving researcher's competence on a variety of issues, bestowing prestige and authority on those who have good perspectives on long-term risks, improving researcher's networks, and training junior researchers. However, often it is best to focus on the product model of research anyway, and get these benefits as a side effect.

To quote the author: "I think there is a lot of useful work that can be done in advance, but most of the work involves us building our competence, capacity, and credibility, so that when the time comes, we are in position and ready to formulate a plan. [...] Investments we make today should increase our competence in relevant domains, our capacity to grow and engage effectively, and the intellectual credibility and policy influence of competent experts."


AGI Strategy - List of Resources

Exactly what it sounds like.


I found the paper to be quite compelling, especially the comments on the human-in-the-loop solution. This feels relevant to problems in technical AI alignment, though I'm not exactly sure how. One question that it suggests -- how can we learn human preferences, when the human answers may themselves depend on the AI's actions? Stuart Armstrong has [pointed out](https://agentfoundations.org/item?id=1678) this problem as well.

Technology Roulette: Managing Loss of Control as Many Militaries Pursue Technological Superiority

US policy so far has been to pursue technological superiority in order to stay ahead of its adversaries and to prevent conflict through deterrence. This paper argues that policymakers should shift some attention to preparing for other risks, such as accidents, emergent effects, sabotage and proliferation (where other actors get and use the technology, without the same safety standards as the US). There were several interesting sections, but the one I was particularly interested in was the section arguing that keeping a human in the loop would not be sufficient. In military situations, decisions must often be made in time-sensitive, high-stress situations, and in such scenarios humans are not very good at making decisions. For example, if an AI system detects an incoming missile, it must autonomously aim and fire to prevent the missile from hitting its target -- there is not enough time for a human to be in the loop. The biggest issue though is that while a human may be part of the decision-making process, they are reliant on various machine readings and calculations in order to reach their decision, and so a human in the loop doesn't provide an independent check on the answer, and so is of limited utility. And as AI systems get better, humans will become less useful for checking the AI's decisions, making this a temporary solution at best.


I am very impressed by the number and diversity of experts that contributed to this report. I think it is quite valuable to get people with such different backgrounds and areas of expertise to collaborate on how we should be using AI ahead of time. I was also pleasantly surprised to hear that there was broad international agreement on principles so far, particularly given an overall political trend against global institutions that has occurred recently. I’m definitely interested to know what the key factors were in managing that and how we can make sure these things continue.

Another piece that jumped out at me is the overlap between longer-term issues of safety and shorter-term issues of bias and privacy. For technical safety work, I think the problems are largely distinct and it is important for safety researchers to remain focused on solving problems with major long-term consequences. However, in the governance context, the problems seem to have much more in common and require many similar institutions / processes to address. So I hope that these communities continue to work together and learn from each other.

AI Governance in 2019 - A Year in Review: Observations from 50 Global Experts

This report contains short essays from 50 experts reviewing progress in AI governance. I'll describe a few themes here rather than try to summarize each essay.

The first is a strong emphasis on issues of bias, privacy, deception, and safety. Bias can occur both due to biases of programmers designing algorithms as well as bias that exists in the data. Deception includes deepfakes as well as online accounts that impersonate humans, a subset of which were made illegal in California this year.

The benefit of international collaborations and conferences and getting broad agreement from many stakeholders both in government and companies was frequently highlighted throughout. One example is the [OECD Principles on AI](https://www.oecd.org/going-digital/ai/principles/), which were later adopted by the G20 including both the US and China, but there were many working groups and committees organized as well, both within industry and governments.

The other shift in 2019 was moving from broad principles towards more specific sets of requirements and policy decisions. The principles agreed to have been quite similar, but the specific implementations vary significantly by country. There were individual essays describing the regional challenges in Europe, the UK, Japan, Singapore, India, and East Asia. Many essays also highlighted the debate around <@publication norms@>(@GPT-2: 1.5B Release@), which garnered a lot of attention in 2019 following OpenAI's staged release of GPT-2.


It would be exciting to see policymakers and AI developers experimenting with the proposed mechanisms, not because I am confident that all of these mechanisms will be useful for incentivising safer AI development, but because trying solutions and observing their specific shortcomings is useful for coming up with better solutions, and trying things early gives us more time to improve and iterate. However, the danger of lock-in is real as well: if the mechanisms won't be iterated on, delaying implementation until everything is watertight could be the better option. On the level of concrete mechanisms, regular red teaming exercises and more research on interpretability seem especially useful for safety, as common interaction with failure modes of AI systems seems to make safety issues more salient.

**Rohin's opinion:** I love seeing these huge collaborations that simply enumerate possibilities for achieving some goal (the previous one, also organized by Miles, is the [Malicious use of AI paper](https://arxiv.org/abs/1802.07228)). It gives me much more confidence that the list is in some sense "exhaustive"; any items not on the list seem more likely to have been intentionally excluded (as opposed to the authors failing to think of those items).

That said, these mechanisms are still only one type of way that you could use to build trust -- in practice, when I trust people, it's because I think they also want to do the things I want them to do (whether because of external incentives or their intrinsic goals). I wonder how much this kind of trust building can be done with AI. For example, one story you could tell is that by producing compelling evidence that a particular algorithm is likely to lead to an existential catastrophe, you can build trust that no one will use that algorithm, at least as long as you believe that everyone strongly wants to avoid an existential catastrophe (and this outweighs any benefits of running the algorithm).

Paper: Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims

Improving Verifiability in AI Development

Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, Tegan Maharaj, Pang Wei Koh, Sara Hooker, Jade Leung, Andrew Trask, Emma Bluemke, Jonathan Lebensbold, Cullen O'Keefe, Mark Koren, Théo Ryffel, JB Rubinovitz, Tamay Besiroglu, Federica Carugati, Jack Clark, Peter Eckersley, Sarah de Haas, Maritza Johnson, Ben Laurie, Alex Ingerman, Igor Krawczuk, Amanda Askell, Rosario Cammarota, Andrew Lohn, David Krueger, Charlotte Stix, Peter Henderson, Logan Graham, Carina Prunkl, Bianca Martin, Elizabeth Seger, Noa Zilberman, Seán Ó hÉigeartaigh, Frens Kroeger, Girish Sastry, Rebecca Kagan, Adrian Weller, Brian Tse, Elizabeth Barnes, Allan Dafoe, Paul Scharre, Ariel Herbert-Voss, Martijn Rasser, Shagun Sodhani, Carrick Flynn, Thomas Krendl Gilbert, Lisa Dyer, Saif Khan, Yoshua Bengio, Markus Anderljung

This multi-stakeholder report by authors from 30 different organizations proposes mechanisms to help with making claims about AI systems easier to verify. Despite being far from a complete solution to responsible AI development, verifiable claims can enable the public to hold developers accountable to comply with their stated ethics principles and allow developers to build trust by providing hard evidence about safety and fairness of their system. Better mechanisms for making such claims could also: i) help with the regulation of AI systems, ii) improve safety by counterbalancing competitive pressures to cut corners, and iii) help independent outside parties to assess the risk posed by specific applications.

The proposed mechanisms cluster into three classes: institutional, software, and hardware. Institutional mechanisms act on the incentives AI developers face. These include **third party auditing** of AI systems, for which a task force investigating different options would be helpful, and the **publication of incidents**, to provide evidence that incidents are taken seriously and prevent others from repeating the same mistakes. Other approaches are broadly analogous to adversarial training: collaborative **red teaming exercises** help to explore risks, and **bias and safety bounties** incentivize outsiders to seek out and report problems.

Software mechanisms include **audit trails** that are used in many safety-critical applications in other industries, better **interpretability** to help with risk assessment and auditing, as well as better tools and standardization for **privacy-preserving machine learning**. The proposed hardware mechanisms are **secure hardware for machine learning**, which requires additional investment as machine learning often uses specialized hardware such that progress in the security of commodity hardware cannot be directly leveraged, **high-precision compute measurement** to help with verifying claims about how many computational resources were used for a particular project, and **compute support for academia** to allow academic researchers to better scrutinize claims made by the AI industry.


While there are certainly major benefits to the Windfall Clause in the case of an actual windfall, it seems to me like there are benefits even when windfalls do not occur (a point mentioned but not emphasized in the full report). For example, in a world in which everyone has agreed to the Windfall Clause, the incentives to "win an economic race" decrease: even if it is possible for e.g. one company to "win" via a monopoly on AI, at least a portion of their "winnings" must be distributed to everyone else, plausibly decreasing incentives to race, and increasing the likelihood that companies pay attention to safety. (This of course assumes that the clause remains binding even after "winning", which is not obviously true.)

The Windfall Clause: Distributing the Benefits of AI

Cullen O’Keefe, Peter Cihon, Ben Garfinkel, Carrick Flynn, Jade Leung, Allan Dafoe

The Windfall Clause is a proposed policy lever for improving outcomes from transformative AI. Corporations can voluntarily agree to be bound by the clause, in which case they must donate some proportion of _windfall profits_ (profits in excess of e.g. 1% of world GDP) for the benefit of humanity. Since such a scenario is exceedingly unlikely, it can be in corporations' interests to be bound by this clause, in order to reap the benefits of improved public relations. If the scenario actually occurs, we can then use the donations to solve many societal problems that would likely arise, e.g. job loss, inequality, etc.


This is a remarkably straightforward and clear analysis, and is way better than any analysis I've seen done by the AI community, which is quite a shame, given how much time AI researchers have spent thinking about publication norms. (Though note that I don't follow this space closely, and so might have missed previous good work on publication norms.) As a comparison, the only conclusion that <@When Is It Appropriate to Publish High-Stakes AI Research?@> came to was that whatever the publication norms are, they should be standardized across the AI community.

The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?

Since <@GPT-2@>(@Better Language Models and Their Implications@), the AI research community has wrestled with the question of publication of research with malicious applications. On the one hand, publishing such research makes it more likely that those malicious applications arise in reality, but on the other hand, it also allows defenses against the application to be developed. The core of the question is what the _offense-defense balance_ of AI research looks like.

In particular, publication is particularly good if attackers are likely to independently develop the knowledge, or would find it hard to translate the research into a real-world attack, or if defenders will put in a lot of effort to finding a solution, and such a solution is likely to be found and deployed. A canonical example is computer security: once a vulnerability is found, it is usually quite easy to develop a patch that fixes the vulnerability, and such patches can be deployed relatively easily via automatic updates. As a result, in computer security, the default is to publicly disclose vulnerabilities after giving vendors some time to develop and deploy a patch.

Under the opposite conditions, where attackers are likely to be able to use the research to create a real-world attack, or where defenders would find it hard to find and deploy a good solution, it is better to keep the research secret. For example, in biorisks such as the risk of an engineered pandemic, solutions are not necessarily easy to find and/or deploy, and so it seems better to avoid making public the knowledge of how to create a novel virus.

The paper argues that relative to computer security (the default comparison for many AI researchers), publication in AI is more likely to be net negative (specifically from a security standpoint, ignoring beneficial applications of the research), since solutions must often be social (as in e.g. fake news) which are harder to deploy, and publication seems more likely to counterfactually educate attackers rather than defenders (since the defenders are big companies that already have a lot of expertise).


I found it quite surprising that the paper mentions a lack of academic consensus about whether or not offensive advantage is destabilizing. Assuming that it is, OD-scaling might provide a silver lining concerning cybersecurity, provided things can be scaled up sufficiently. These kinds of dynamics also seem to put a natural ceiling on arms races: above a certain threshold, gains in capabilities provide advantage to both sides such that resources are better invested elsewhere.

How does the offense-defense balance scale?

The offense-defense balance that characterises how easy it is to successfully attack others can affect what kinds of conflicts break out and how often that happens. This paper analyses how growing capabilities on both sides affect that balance. For example, consider an idealized model of cyber defense with a fixed set of vulnerabilities that are discovered independently by attackers and defenders. The attacker will initially be able to use almost all of the vulnerabilities they found. This is because, with only a small percentage of vulnerabilities discovered by both sides, the defender is unlikely to have found the same ones as the attacker. Marginal increases of the defender's capabilities are unlikely to uncover vulnerabilities used by the attacker in this regime, such that attacks become easier as both sides invest resources. Once most vulnerabilities have been found by both sides, this effect reverses as marginal investments by the attacker become unlikely to uncover vulnerabilities the defender has not fixed yet.

This pattern, where increasingly growing capabilities first favour offense but lead to defensive stability in the long run, dubbed **OD-scaling** seems to be common and can be expected to be found whenever there are **multiple attack vectors**, the attacker only needs to break through on some of them and the defender enjoys **local defense superiority**, meaning that with sufficient coverage by the defender for a given attack vector, it is almost impossible for the attacker to break through.

Because the use of digital and AI systems can be scaled up quickly, scale-dependent shifts of the offense-defense balance are going to increase in importance as these systems become ubiquitous.


Given that much of current AI progress is being driven by increases in computation power, it seems likely to me that companies will soon become more significant players in the AI space. As a result, I appreciate that this paper tries to determine what we can do now to make sure that the competitive landscape is conducive to taking proper safety precautions. I do, however, believe that the single step cooperate-defect game which they use to come up with their factors seems like a very simple model for what will be a very complex system of interactions. For example, AI development will take place over time, and it is likely that the same companies will continue to interact with one another. Iterated games have very different dynamics, and I hope that future work will explore how this would affect their current recommendations, and whether it would yield new approaches to incentivizing cooperation.

The Role of Cooperation in Responsible AI Development

Why Responsible AI Development Needs Cooperation on Safety

Amanda Askell, Miles Brundage, Jack Clark, Gillian Hadfield

AI systems are increasingly being developed by companies, and as such it is important to understand how competition will affect the safety and robustness of these systems. This paper models companies as agents engaging in a cooperate-defect game, where cooperation represents responsible development, and defection represents a failure to develop responsibly. This model yields five factors that increase the likelihood of companies cooperating on safety. Ideally, companies will have **high trust** that others cooperate on safety, large benefits from mutual cooperation (**shared upside**), large costs from mutual defection (**shared downside**), not much incentive to defect when others cooperate (**low advantage**), and not be harmed too much if others defect when they cooperate (**low exposure**).

They then suggest four concrete strategies that can help improve norms today. First, companies should help promote accurate beliefs about the benefits of safety. Second, companies should collaborate on research and engineering. Third, companies should be transparent and allow for proper oversight and feedback. Fourth, the community should incentivize adhering to high safety standards by rewarding safety work and penalizing unsafe behavior.


AI Alignment Podcast: China’s AI Superpower Dream

See also <@these@>(@Rationally Speaking #231 - Helen Toner on "Misconceptions about China and artificial intelligence"@) <@three@>(@The new 30-person research group in DC investigating how emerging technologies could affect national security@) <@podcasts@>(@FLI Podcast: Beyond the Arms Race Narrative: AI and China@).


Soft takeoff can still lead to decisive strategic advantage

Since there will be an improved version of this post soon, I will summarize it then.


This argument is quite compelling to me under the assumption of human-level AGI systems that can be intent-aligned. Note though that while the development of AGI systems removes principal-agent problems, it doesn't remove issues that arise due to different agents having different (non-value-related) information.

The argument probably doesn't hold with <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@), where each AI service is optimized for a particular task, since there would be principal-agent problems between services.

It seems like the argument should mainly make us more worried about stable authoritarian regimes: the main effect based on this argument is a centralization of power in the hands of the AGI's overseers. This is less likely to happen with companies, because we have institutions that prevent companies from gaining too much power, though perhaps competition between countries could weaken such institutions. It could happen with government, but if long-term governmental power still rests with the people via democracy, that seems okay. So the risky situation seems to be when the government gains power, and the people no longer have effective control over government. (This would include scenarios with e.g. a government that has sufficiently good AI-fueled propaganda that they always win elections, regardless of whether their governing is actually good.)

AGI will drastically increase economies of scale

Economies of scale would normally mean that companies would keep growing larger and larger. With human employees, the coordination costs grow superlinearly, which ends up limiting the size to which a company can grow. However, with the advent of AGI, many of these coordination costs will be removed. If we can align AGIs to particular humans, then a corporation run by AGIs aligned to a single human would at least avoid principal-agent costs. As a result, the economies of scale would dominate, and companies would grow much larger, leading to more centralization.


Beijing publishes AI ethical standards, calls for int'l cooperation

Beijing AI Principles

These principles are a collaboration between Chinese academia and industry, and hit upon many of the problems surrounding AI discussed today, including fairness, accountability, transparency, diversity, job automation, responsibility, ethics, etc. Notably for long-termists, it specifically mentions control risks, AGI, superintelligence, and AI races, and calls for international collaboration in AI governance.


Stable Agreements in Turbulent Times

On the one hand we would like actors to be able to cooperate before the development of AGI by entering into binding agreements, but on the other hand such agreements are often unpalatable and hard to write because there is a lot of uncertainty, indeterminacy and unfamiliarity with the consequences of developing powerful AI systems. This makes it very hard to be confident that any given agreement is actually net positive for a given actor. The key point of this report is that we can strike a balance between these two extremes by agreeing pre-AGI to be bound by decisions that are made post-AGI with the benefit of increased knowledge. It examines five tools for this purpose: options, impossibility doctrines, contractual standards, renegotiation, and third-party resolution.


I find this talk to be eminently reasonable throughout. It highlights a concerning lack of public high-quality engagement with the fundamental ideas in AI safety over the last few years, relative to the growth of the field as a whole (although note that in the past few months this has been changing, with three excellent sequences released on the Alignment Forum, plus Drexler's technical report). This is something which motivates me to spend a fair amount of time writing about and discussing such ideas.

One nitpick: I dislike the use of "accidents" as an umbrella term for AIs behaving in harmful ways unintended by their creators, since it's misleading to describe deliberately adversarial behaviour as an "accident" (although note that this is not specific to Ben's talk, since the terminology has been in use at least since the Concrete problems paper).

How Sure are we about this AI Stuff?

Ben outlines four broad arguments for prioritising work on superintelligent AGI: that AI will have a big influence over the long-term future, and more specifically that it might cause instability, lock-in or large-scale "accidents". He notes the drawbacks of each line of argument. In particular, the "AI is a big deal" argument doesn't show that we have useful leverage over outcomes (compare a Victorian trying to improve the long-term effects of the industrial revolution). He claims that the next two arguments have simply not been researched thoroughly enough to draw any conclusions. And while the argument from accidents has been made by Bostrom and Yudkowsky, there hasn't been sufficient elaboration or criticism of it, especially in light of the recent rise of deep learning, which reframes many ideas in AI.


AI Index 2018 Report

Yoav Shoham, Raymond Perrault,Erik Brynjolfsson, Jack Clark, James Manyika, Juan Carlos Niebles,Terah Lyons, John Etchemendy, Barbara Grosz

Lots of data about AI. The report highlights how AI is global, the particular improvement in natural language understanding over the last year, and the limited gender diversity in the classroom. We also see the expected trend of huge growth in AI, both in terms of interest in the field as well as in performance metrics.


AI Now 2018 Report

Meredith Whittaker, Kate Crawford, Roel Dobbe, Genevieve Fried, Elizabeth Kaziunas, Varoon Mathur, Sarah Myers West, Rashida Richardson, Jason Schultz, Oscar Schwartz


There aren't enough details on the technical research for me to say anything useful about it. I'm broadly in support of deconfusion but am either less optimistic on the tractability of deconfusion, or more optimistic on the possibility of success with our current notions (probably both). Keeping research unpublished-by-default seems reasonable to me given the MIRI viewpoint for the reasons they talk about, though I haven't thought about it much. See also [Import AI](https://jack-clark.net/2018/11/26/import-ai-122-google-obtains-new-imagenet-state-of-the-art-with-gpipe-drone-learns-to-land-more-effectively-than-pd-controller-policy-and-facebook-releases-its-cherrypi-starcraft-bot/).

MIRI 2018 Update: Our New Research Directions

This post gives a high-level overview of the new research directions that MIRI is pursuing with the goal of deconfusion, a discussion of why deconfusion is so important to them, an explanation of why MIRI is now planning to leave research unpublished by default, and a case for software engineers to join their team.


The model in [Racing to the precipice](https://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf) had the unintuitive conclusion that if teams have _more_ information (i.e. they know their own or other’s capabilities), then we become _less_ safe, which puzzled me for a while. Their explanation is that with maximal information, the top team takes as much risk as necessary in order to guarantee that they beat the second team, which can be quite a lot of risk if the two teams are close. While this is true, the explanation from this post is more satisfying -- since the model has a discontinuity that rewards taking on risk, anything that removes the discontinuity and makes it more continuous will likely improve the prospects for safety, such as not having full information. I claim that in reality these discontinuities mostly don't exist, since (1) we're uncertain about who will win and (2) we will probably have a multipolar scenario where even if you aren't first-to-market you can still capture a lot of value. This suggests that it likely isn't a problem for teams to have more information about each other on the margin.

That said, these models are still very simplistic, and I mainly try to derive qualitative conclusions from them that my intuition agrees with in hindsight.

Racing to the precipice: a model of artificial intelligence development

AI development incentive gradients are not uniformly terrible

This post considers a model of AI development somewhat similar to the one in [Racing to the precipice](https://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf) paper. It notes that under this model, assuming perfect information, the utility curves for each player are _discontinuous_. Specifically, the models predict deterministically that the player that spent the most on something (typically AI capabilities) is the one that "wins" the race (i.e. builds AGI), and so there is a discontinuity at the point where the players are spending equal amounts of money. This results in players fighting as hard as possible to be on the right side of the discontinuity, which suggests that they will skimp on safety. However, in practice, there will be some uncertainty about which player wins, even if you know exactly how much each is spending, and this removes the discontinuity. The resulting model predicts more investment in safety, since buying expected utility through safety now looks better than increasing the probability of winning the race (whereas before, it was compared against changing from definitely losing the race to definitely winning the race).


This is an important paper which I hope will lead to much more analysis of these questions.

The Vulnerable World Hypothesis

Bostrom considers the possibility "that there is some level of technology at which civilization almost certainly gets destroyed unless quite extraordinary and historically unprecedented degrees of preventive policing and/or global governance are implemented." We were lucky, for example, that starting a nuclear chain reaction required difficult-to-obtain plutonium or uranium, instead of easily-available materials. In the latter case, our civilisation would probably have fallen apart, because it was (and still is) in the "semi-anarchic default condition": we have limited capacity for preventative policing or global governence, and people have a diverse range of motivations, many selfish and some destructive. Bostrom identifies four types of vulnerability which vary by how easily and widely the dangerous technology can be produced, how predictable its effects are, and how strong the incentives to use it are. He also idenitifies four possible ways of stabilising the situation: restrict technological development, influence people's motivations, establish effective preventative policing, and establish effective global governance. He argues that the latter two are more promising in this context, although they increase the risks of totalitarianism. Note that Bostrom doesn't take a strong stance on whether the vulnerable world hypothesis is true, although he claims that it's unjustifiable to have high credence in its falsity.


The Future of Surveillance

While we often think of there being a privacy-security tradeoff and an accountability-security tradeoff with surveillance, advances in AI and cryptography can make advances on the Pareto frontier. For example, automated systems could surveil many people but only report a few suspicious cases to humans, or they could be used to redact sensitive information (eg. by blurring faces), both of which improve privacy and security significantly compared to the status quo. Similarly, automated ML systems can be applied consistently to every person, can enable collection of good statistics (eg. false positive rates), and are more interpretable than a human making a judgment call, all of which improve accountability.


Truthful AI: Developing and governing AI that does not lie

Owain Evans*, Owen Cotton-Barratt*, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, William Saunders

This paper argues that we should develop both the technical capabilities and the governance mechanisms necessary to ensure that AI systems can be made _truthful_. We will primarily think about conversational AI systems here (so not, say, AlphaFold).

Some key terms:
1. An AI system is **honest** if it only makes statements that it actually believes. (This requires you to have some way of ascribing beliefs to the system.) In contrast, **truthfulness** only checks if statements correspond to reality, without making any claims about the AI system's beliefs.
2. An AI system is **broadly truthful** if it doesn't lie, volunteers all the relevant information it knows, is well-calibrated and knows the limits of its information, etc.
3. An AI system is **narrowly truthful** if it avoids making **negligent suspected-falsehoods**. These are statements that can feasibly be determined by the AI system to be unacceptably likely to be false. Importantly, a narrowly truthful AI is not required to make contentful statements, it can express uncertainty or refuse to answer.

This paper argues for narrow truthfulness as the appropriate standard. Broad truthfulness is not very precisely defined, making it challenging to coordinate on.  Honesty does not give us the guarantees we want: in settings in which it is advantageous to say false things, AI systems might end up being honest but **deluded**. They would honestly report their beliefs, but those beliefs might be false.

Narrow truthfulness is still a much stronger standard than we impose upon humans. This is desirable because (1) AI systems need not be constrained by social norms the way humans are; consequently they need stronger standards, and (2) it may be less costly to enforce that AI systems are narrowly truthful than to enforce that humans are narrowly truthful, so a higher standard is more feasible.

Evaluating the (narrow) truthfulness of a model is non-trivial. There are two parts: first, determining whether a given statement is unacceptably likely to be false, and second, determining whether the model was negligent in uttering such a statement. The former could be done by having human processes that study a wide range of information and determine whether a given statement is unacceptably likely to be false. In addition to all of the usual concerns about the challenges of evaluating a model that might know more than you, there is also the challenge that it is not clear exactly what counts as "unacceptably likely to be false". For example, if a model utters a false statement but expresses low confidence, how should that be rated? The second part, determining negligence, needs to account for the fact that the AI system might not have had all the necessary information, or that it might not have been capable enough to come to the correct conclusion. One way of handling this is to compare the AI system to other AI systems built in a similar fashion.

How might narrow truthfulness be useful? One nice thing it enables is **truthfulness amplification**, in which we can amplify properties of a model by asking a web of related questions and combining the answers appropriately. For example, if we are concerned that the AI system is deceiving us on just this question, we could ask it whether it is deceiving us, or whether an investigation into its statement would conclude that it was deceptive. As another example, if we are worried that the AI system is making a mistake on some question where its statement isn't _obviously_ false, we can ask it about its evidence for its position and how strong the evidence is (where false statements are more likely to be negligently false).

Section 3 is devoted to the potential benefits and costs if we successfully ensure that AI systems are narrowly truthful, with the conclusion that the costs are small relative to the benefits and can be partially mitigated. Section 6 discusses other potential benefits and costs if we attempt to create truthfulness standards to ensure the AI systems are narrowly truthful. (For example, we might try to create a truthfulness standard but instead create an institution that makes sure that AI systems follow a particular agenda (by only rating as true the statements that are consistent with that agenda). Section 4 talks about the governance mechanisms we might use to implement a truthfulness standard. Section 5 describes potential approaches for building truthful AI systems. As I mentioned in the highlighted post, these techniques are general alignment techniques that have been specialized for truthful AI.


Decoupling deliberation from competition

Under a [longtermist](https://forum.effectivealtruism.org/tag/longtermism) lens, one problem to worry about is that even after building AI systems, humans will spend more time competing with each other rather than figuring out what they want, which may then lead to their values changing in an undesirable way. For example, we may have powerful persuasion technology that everyone uses to persuade people to their line of thinking; it seems bad if humanity's values are determined by a mix of effective persuasion tools, especially if persuasion significantly diverges from truth-seeking.

One solution to this is to coordinate to _pause_ competition while we deliberate on what we want. However, this seems rather hard to implement. Instead, we can at least try to _decouple_ competition from deliberation, by having AI systems acquire <@flexible influence@>(@The strategy-stealing assumption@) on our behalf (competition), and having humans separately thinking about what they want (deliberation). As long as the AI systems are competent enough to shield the humans from the competition, the results of the deliberation shouldn't depend too much on competition, thus achieving the desired decoupling.

The post has a bunch of additional concrete details on what could go wrong with such a plan that I won't get into here.


A personal take on longtermist AI governance

We've <@previously seen@>(@Our AI governance grantmaking so far@) that Open Philanthropy struggles to find intermediate goals in AI governance that seem robustly good to pursue from a longtermist perspective. (If you aren't familiar with longtermism, you probably want to skip to the next summary.) In this personal post, the author suggests that there are three key bottlenecks driving this:

1. There are very few longtermists in the world; those that do exist often don't have the specific interests, skills, and experience needed for AI governance work. We could try to get others to work on relevant problems, but:
2. We don't have the strategic clarity and forecasting ability to know which intermediate goals are important (or even net positive). Maybe we could get people to help us figure out the strategic picture? Unfortunately:
3. It's difficult to define and scope research projects that can help clarify which intermediate goals are worth pursuing when done by people who are not themselves thinking about the issues from a longtermist perspective.

Given these bottlenecks, the author offers the following career advice for those who hope to do work from a longtermist perspective in AI governance:

1. Career decisions should be especially influenced by the value of experimentation, learning, [aptitude development](https://forum.effectivealtruism.org/posts/bud2ssJLQ33pSemKH/my-current-impressions-on-career-choice-for-longtermists), and career capital.
2. Prioritize future impact, for example by building credentials to influence a 1-20 year "crunch time" period. (But make sure to keep studying and thinking about how to create that future impact.)
3. Work on building the field, especially with an eye to reducing bottleneck #1. (See e.g. [here](https://forum.effectivealtruism.org/posts/EEtTQkFKRwLniXkQm/open-philanthropy-is-seeking-proposals-for-outreach-projects).)
4. Try to reduce bottleneck #2 by doing research that increases strategic clarity, though note that many people have tried this and it doesn't seem like the situation has improved very much.


Corporate Governance of Artificial Intelligence in the Public Interest

Peter Cihon, Jonas Schuett, Seth D. Baum

This paper is a broad overview of corporate governance of AI, where by corporate governance we mean "anything that affects how AI is governed within corporations" (a much broader category than the governance that is done by corporations about AI). The authors identify nine primary groups of actors that can influence corporate governance and give many examples of how such actors have affected AI governance in the past. The nine groups are managers, workers, investors, corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, and governments.

Since the paper is primarily a large set of examples along with pointers to other literature on the topic, I'm not going to summarize it in more detail here, though I did find many of the examples interesting (and would dive into them further if time was not so scarce).


I think the headline was chosen by the editor and not the author: the AI arms race 'debunking' is less than a fourth of the whole article, and it's not even an important beat of the piece; instead, the article is about how use of technology/AI/deep learning for military applications in multipolar geopolitics *can actually result* in arms-race-style dynamics and tangible risks.

Even so, I'm not convinced that the traditional definition of 'arms race' isn't met. The author invokes percentage _growth_ in military spending of more than 10% over the previous year as a qualifying criterion for an arms race, but then compares this with the actual spending of 0.7% of the US military budget on AI in 2020 to make their case that there is no arms race. These two are not comparable; at the very least, we would need to know the actual spending on AI by the military across two years to see at what rate this spending changed, and whether or not it then qualifies to be an arms race.

Debunking the AI Arms Race Theory

This article, published recently in the Texas National Security Review, argues that various national trends of military spending on AI do not meet the traditional definition of an 'arms race'. However, the current situation can be termed a _security dilemma_, a "more generalized competitive dynamic between states." The article identifies two ways in which race-style dynamics in AI competition towards the aims of national security might create new risks: (i) a need for increasingly rapid decision-making might leave humans with diminished control or 'out of the loop'; and (ii) the pressure to quickly improve military AI capabilities could result in sacrificing supplementary goals like robustness and reliability, leading to unsafe systems being deployed.

The article offers the following strategies as panaceas to such dynamics. Competing nations should institute strong internal processes to ensure their systems are robust and secure, and that human control can be maintained. Further, nations should encourage other countries to take similar steps to mitigate these risks within their own militaries. Finally, nations should cooperate in regulating the conduct of war to avoid mutual harm. It concludes after citing several sources that advocate for the US to adopt these strategies.


The most interesting part of this report for me was the example outputs that the authors gave in the report, which showcase how GPT-3 can be used to “argue” in support or against a variety of topics, in a manner meant to be persuasive to a specific group of people (for example, arguing to Jews that they should vote Democratic / vote Republican / not vote).

(I put “argue” in quotation marks because the outputs hardly feel like what I would call “arguments” for a position, instead simply appealing to something the group agrees with and stating with barely any argument / evidence that this implies the position to be argued for. However, I also have the same critique of most “arguments” that I see on Twitter -- I don’t think I could distinguish the GPT-3 generated arguments from real human tweets.)

Truth, Lies, and Automation: How Language Models Could Change Disinformation

Ben Buchanan, Andrew Lohn, Micah Musser, Katerina Sedova

Ever since the publication of <@GPT-2@>(@Better Language Models and Their Implications@), the research community has worried about the use of such language models for disinformation campaigns. Disinformation campaigns have happened before: Russia produced thousands of pieces of such content leading up to the 2016 US presidential election. That campaign used large numbers of human workers. Could a future campaign become significantly more effective through the use of large language models?

This report notes that for this threat model, it is primarily worrying if GPT-3 can be used to enable significantly _better_ results, because the monetary cost of hiring humans is not typically a bottleneck for major actors. While GPT-3 by itself is not likely to achieve this, perhaps it can serve as an effective tool for humans, such that the human-machine team can get better results than either one individually.

The authors perform several tests of their own to establish a lower bound on how well human-machine teams can perform currently. They investigate six types of disinformation tasks and find that either GPT-3 can do them easily, or only some human effort is needed to get results that are perceived as high quality by humans, suggesting that this could be a real risk. Unfortunately, it is hard to tell what aspects are _actually_ important for successful disinformation, and this was not something they could ethically check, so it is hard to draw confident conclusions from the study about whether GPT-3 would be useful for disinformation campaigns in practice. (Although their one study on Mechanical Turk did find that GPT-3-generated arguments on international issues like sanctions on China were found to be persuasive and led to significant changes in the proportion of people with the given position.)

One particularly worrying aspect is that the authors found it easier to get GPT-3 to generate extremist content because providing an extremist headline makes it easy to "locate" the appropriate tone and style; whereas with a more moderate headline, GPT-3 might not correctly infer the desired tone or style because the moderate headline could be consistent with lots of tones and styles.


I enjoyed reading this paper and the case studies provided ample context and details to further flesh out the lessons. While the paper might be a bit broad and not entirely AI-specific, I do recommend reading it if you are interested in international cooperation around AI.

International Control of Powerful Technology: Lessons from the Baruch Plan for Nuclear Weapons

This paper explores the analogy between early attempts at the international control of nuclear technology and the international control of today's emerging powerful technologies such as AI. While nuclear technology was perceived as very powerful, and many considered it an existential risk, there was also substantial uncertainty about its impacts. In addition, nuclear technology relied on rapid scientific development and engendered national competition, negotiation, and arms race dynamics. Lastly, there was a lot of policy debate about the political and ethical aspects of nuclear technology, including discussions on international governance ("international control").

The authors provide ten lessons from the history of nuclear control and support them with evidence from various case studies:
 - Radical proposals might be discussed seriously or even adapted as official policy in the light of very disruptive technologies and upheavals in international politics.
 - Actors' support or opposition to international control can be influenced by opportunism and their position might shift over time. Thus, it is particularly important to build broad coalitions.
 - In particular, schemes for international governance are sometimes supported by "realists" focussed on power politics, but such support is often fickle.
 - Secrecy around powerful technologies play an important role and can be abused by actors controlling information flow within a country. Secrecy should only be expanded with care and policymakers need to ensure they are informed by a wide range of perspectives.
 - Public opinion has an important effect on debates about international control, and elites benefit from trying to shape it.
 - Technical experts can influence policy to be more effective and cooperative, but they need to understand the political landscape.
 - Policymaking often does not involve grand strategy; instead, it can better be described as muddling through, even in the realm of international control.
 - International control is difficult, and it is unclear whether strategic obstacles can be circumvented.
 - International cooperation can require countries to take substantial risk. It is important for international cooperation advocates to understand these risks and point out avenues for mitigation.
 - Even maximally destructive solutions like preventive strikes can get political traction.

However, there are also important differences between nuclear technology and AI or other emerging technologies that have to be kept in mind: First, AI involves less secrecy and relies on the private sector more strongly. In addition,  AI is already used around the world, and its use is harder to detect, such that proliferation might be harder to prevent. Lastly, the dangers of AI are less obvious, accidents might be harder to manage, and the strategic advantage from advanced AI might not plateau, as it has for nuclear weapons once second-strike capabilities were achieved. More broadly, the historical context regarding nuclear technology was influenced by WW2, the visceral examples of Hiroshima and Nagasaki, as well as a less globalized world with stronger zero-sum dynamics between superpowers.


National Security Commission on Artificial Intelligence

NSCAI Final Report

Eric Schmidt, Robert Work, Safra Catz, Eric Horvitz, Steve Chien, Andrew Jassy, Mignon Clyburn, Gilman Louie, Chris Darby, William Mark, Kenneth Ford, Jason Matheny, José-Marie Griffiths, Katharina McFarland, Andrew Moore

In the US, the National Security Commission on AI released their report to Congress. The [full pdf](https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf) is over 750 pages long, so I have not read it myself, and instead I'm adding in some commentary from others. In their [newsletter](https://cset.georgetown.edu/newsletters/), CSET says that highlights include:

- A warning that the U.S. military could be at a competitive disadvantage within the next decade if it does not accelerate its AI adoption. The report recommends laying the foundation for widespread AI integration by 2025, comprising a DOD-wide digital ecosystem, a technically literate workforce, and more efficient business practices aided by AI.
- A recommendation that the White House establish a new "Technology Competitiveness Council," led by the vice president, to develop a comprehensive technology strategy and oversee its implementation.
- A recommendation that the U.S. military explore using autonomous weapons systems, provided their use is authorized by human operators.
- A proposal to establish a new Digital Service Academy and a civilian National Reserve to cultivate domestic AI talent.
- A call to provide $35 billion in federal investment and incentives for domestic semiconductor manufacturing.
- A recommendation to double non-defense AI R&D funding annually until it reaches $32 billion per year, and to triple the number of National AI Research Institutes.
- A call for reformed export controls, coordinated with allies, on key technologies such as high-end semiconductor manufacturing equipment.
- A recommendation that Congress pass a second National Defense Education Act and reform the U.S. immigration system to attract and retain AI students and workers from abroad.

While none of the report's recommendations are legally binding, it has [reportedly been well-received by key members of both parties](https://apnews.com/article/ai-panel-urges-us-boost-tech-skills-95b210543d4a42bd6cd5347a46cb74d6).

Matthew van der Merwe also summarizes the recommendations in [Import AI](https://jack-clark.net/2021/03/08/import-ai-239-china-trains-a-massive-10b-model-vicarious-does-pickplace-the-gchq-publishes-some-of-its-thoughts-on-ai/); this has a lot of overlap with the CSET summary so I won't copy it here.

Jeff Ding adds in [ChinAI #134](https://chinai.substack.com/p/chinai-134-weaponized-interdependence):
[I]f you make it past the bluster in the beginning — or take it for what it is: obligatory marketing to cater to a DC audience hooked on a narrow vision of national security — there's some smart moderate policy ideas in the report (e.g. chapter 7 on establishing justified confidence in AI systems).

In email correspondence, Jon Rodriguez adds some commentary on the safety implications:

1. The report acknowledges the potential danger of AGI, and specifically calls for value alignment research to take place (pg. 36). To my knowledge, this is one of the first times a leading world government has called for value alignment.
2. The report makes a clear statement that the US prohibits AI from authorizing the launch of nuclear weapons (pg. 98).
3. The report calls for dialogues with China and Russia to ensure that military decisions made by military AI at "machine speed" does not lead to out-of-control conflict escalation which humans would not want (pg. 97).


I agree completely with the first two points. Much of technical safety work has been based around solving currently existing analogs of the alignment problem. Governance does seem to have less of these, so autonomous weapon governance could provide a great opportunity to test and build credibility for AI governance structures. The ability for autonomous weapons to cause catastrophic risk seems hard to argue against. With powerful enough AI, even accidents can pose catastrophic risk, but I would expect military use to only increase those.

For the third point, I agree with the reasons provided, but I think there are also ways in which AWs may reduce the likelihood and severity of war. For instance, currently soldiers bear most of the risk in wars, whereas decision-makers are often protected. Targeted AW attacks may increase the relative risk for those making decisions and thus disincentivize them from declaring war. An equilibrium of AW mutually assured destruction might also be attained if we can find reliable ways to attribute AW attacks and selectively retaliate. I’d be interested to see a more extensive analysis of how these and other factors trade off as I am unsure of the net effect.

The piece that gives me the most doubt that this is an area for the x-risk community to focus on is tractability. An international agreement runs the risk of weakening the states that sign on without slowing the rate of AW development in countries that don’t. Getting all actors to sign on seems intractable to me. As an analogy, nuclear weapons proliferation has been a challenge and nuclear weapons development is much more complex and visible than development of AWs.

**Rohin's opinion:** I particularly liked this piece because it actually made the case for work on autonomous weapons -- I do not see such work as obviously good (see for example [this post](https://forum.effectivealtruism.org/posts/vdqBn65Qaw77MpqXz/on-ai-weapons) that I liked, for the perspective against banning autonomous weapons). I still feel pretty uncertain overall, but I think this post meaningfully moved the debate forward.

Why those who care about catastrophic and existential risk should care about autonomous weapons

This post argues for a focus on autonomous weapons systems (AWs) for three main reasons:

**AWs Provide a Trial Run for AGI governance**. Governance of AWs shares many properties with AGI safety. Preventing an AW arms race would require international cooperation that would provide a chance to understand and improve AI governance institutions. As with any AI system, AWs have the potential to be *effective* without necessarily being aligned with human values, and accidents could quickly lead to deadly consequences. Public opinion and the vast majority of AI researchers oppose AW arms races, so there is an opportunity for global coordination on this issue.

**Some AWs can directly cause catastrophic risk**. Cheap drones could potentially be created at scale that are easy to transport and hard to detect. This could enable an individual to kill many people without the need to convince many others that it is justified. They can discriminate targets better than other WMDs and cause less environmental damage. This has the potential to make war less harmful, but also makes it easier to justify.

**AWs increase the likelihood and severity of conflict** by providing better tools for terrorists and assassins, lowering the threshold for violence between and within states, upsetting the relative power balance of current militaries, and increasing the likelihood of accidental escalation. In particular, AWs that are being used to counter other AWs might intentionally be made hard to understand and predict, and AWs may react to each other at timescales that are too quick for humans to intervene or de-escalate.

An international agreement governing autonomous weapons could help to alleviate the above concerns. In particular, some classes of weapons could be banned, and others could be tracked and subjected to regulations. This would hopefully lead us to an equilibrium where offensive AWs are prohibited, but defended against in a stable way.


I really like the notion of an algorithmic social contract: it much better captures my expectation of how AI systems will be integrated into society. With this vocabulary, I would put technical AI alignment research squarely in the last category, of how we translate fuzzy values that society agrees on into constraints on the AI system’s behavior.

Society-in-the-loop: programming the algorithmic social contract

Earlier in this newsletter we saw arguments that we should not build AI systems that are maximizing "humanity's aggregated preferences". Then how else are we supposed to build AI systems that work well for _society as a whole_, rather than an individual human? When the goal of the system is uncontested (e.g. "don't crash"), we can use human-in-the-loop (HITL) algorithms where the human provides oversight; this paper proposes that for contested goals (e.g. "be fair") we should put society in the loop (SITL), through _algorithmic social contracts_.

What is a social contract? A group of stakeholders with competing interests have a (non-algorithmic) social contract when they "agree" to allow use of force or social pressure to enforce some norm that guards people's rights and punishes violators. For example, we have a social contract against murder, which legitimates the use of force by the government in order to punish violators.

In an algorithmic social contract, the norms by which the AI system operates, and the goals which it pursues, are determined through typical social contracts amongst the group of stakeholders that care about the AI system's impacts. Notably, these goals and norms can change over time, as the stakeholders see what the AI system does. Of course, this all happens on relatively long timescales; more immediate oversight and control of the AI system would have to be done by specific humans who are acting as _delegates_ of the group of stakeholders.

The paper then goes into many open challenges for creating such algorithmic social contracts: How does society figure out what goals the AI system should pursue? How do we deal with externalities and tradeoffs? How can these fuzzy values be translated into constraints on the AI system? It provides an overview of some approaches to these problems.


I’d love to write an opinion on this. Unfortunately, regardless of which “side” of the issue I come down on, if I explained it publicly in this newsletter, I would worry that a journalist would adversarially quote me in a way that I would dislike. (Consider: "Employee confirms DeepMind's efforts to whitewash AI ethics efforts" or "This researcher's fervent defense of Big Tech shows how Google brainwashes its employees".) So I’m not going to say anything here. And just for the record: DeepMind did not tell me not to write an opinion here.

(Am I being too paranoid? Probably, but I've heard of too many horror stories to take the chance. I'm already worrying that someone will quote even this and talk about how Big Tech indoctrinates its employees against virtuous journalists who expose its evils.)

For the sake of transparency about how I make these decisions, if there were only one “side” of the issue that I wouldn’t be willing to express publicly, I would not write an opinion. Also, I doubt that this policy would ever trigger for a technical paper.

The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integrity

Big tech companies fund a lot of academic research, including on AI ethics. This paper points out that we would not trust research on smoking that was funded by tobacco companies: why should AI ethics research be any different? Enough information has now surfaced (through litigation) for us to see that Big Tobacco's actions were clearly unacceptable, but it took years for this to be realized. The same thing could be happening again with Big Tech.

The paper identifies four goals that drive investment into academia by big industries, and argues that these are consistent with the actions of Big Tobacco and Big Tech. First, funding academic research allows companies to present themselves as socially responsible. For example, some researchers have argued that academic or non-profit institutions like the ACLU and MIT do not have any effective power in the Partnership on AI and their membership ends up serving a legitimating function for the companies in the partnership.

Second, companies can influence the events and decisions made by universities. Top conferences in ML receive large sponsorships from companies, and many of the workshops have such sponsorships as well, including ones about AI ethics.

Third, companies can influence the research conducted by individual scientists. The authors studied funding of professors at four top universities, and found that of the cases where they could determine funding, over 52% had been funded by Big Tech, and the number rose to 58% when restricting to those who had published in ethics or fairness. There need not be any explicit pressure for this to be an issue: the implicit threat of loss of funding can be enough to prevent some types of research.

Fourth, companies can discover academics who can be leveraged in other situations. For example, tobacco companies explicitly searched for academics who would testify in favor of the companies at legislative hearings. In Big Tech, there are similar suggestive stories: for example, in one case a professor who had been funded indirectly by Google criticized antitrust scrutiny of Google. They then joined the FTC, and shortly after the FTC dropped their antitrust suit against Google.

The paper concludes with some ideas on how the current situation could be improved.


AI Benefits is effectively asking how we can answer the question of how to do the most good in the future, and as such many of the considerations also come up in effective altruism, especially at the current high level of abstraction. Nonetheless, there are differences in the situation, which will matter: for example, the effective altruism community does not currently need to plan for the situation where they control a majority of the world’s resources; a sufficiently ambitious and optimistic AI company may need to. Such a situation vastly increases the importance of e.g. democratic input, portfolio approaches, and information value. I’m glad that these questions are being tackled now and look forward to seeing more details in the future.

AI Benefits

This sequence of posts investigates _AI Benefits_: how a benefactor can leverage advanced AI systems to benefit humanity. It focuses on what can be done by a single benefactor, outside of what we might think of as the "norm" -- in particular, the sequence ignores benefits that would be provided by default market incentives. This is relevant to OpenAI (where the author works) given their focus on ensuring AI is beneficial to humanity.

Note that AI Benefits is distinct from AI alignment. Sometimes AI alignment is defined broadly enough to encompass AI Benefits, but often it is not, e.g. if the notion of being "aligned" depends on an AI system being aligned with some principal, that would not be AI Benefits, since AI Benefits are meant to accrue to all of humanity. While it is about maximizing well-being by default, it should also have secondary goals of equality, autonomy, democratization, and epistemic modesty.

The obvious approach to AI Benefits is the _direct_ approach: figuring out how to apply advanced AI to directly generate benefits for humanity, e.g. by producing electricity more efficiently to mitigate climate change. However, it is important to also consider the indirect approach of making money using AI, and then donating the surplus to a different organization that can better produce benefits.

Given the massive number of potential ways to benefit humanity and our uncertainty about their efficacy, it is important to have a portfolio approach to AI Benefits, rather than scaling up a single intervention. In addition, since any given intervention will probably primarily benefit some subset of humanity, a portfolio approach should help lead to more equal distribution of benefits.

There are many outstanding questions on how AI Benefits should be done in practice. Should the benefactor pursue a direct or indirect approach? To what extent should they explore potential approaches for generating benefits, relative to exploiting approaches that we know work? Should they generate benefits now, or invest in the ability to generate benefits later? Should they focus on global (supranational) approaches, or allocate resources to each nation that can be used in a manner specialized to their citizens?

There are many questions on the governance side as well. We will presumably want some Benefits Group involving external experts to help distribute benefits optimally. When should such a group get democratic input? How do we evaluate such a group to ensure they are actually benefiting humanity optimally? To what extent will we also need internal governance within the group and benefactor, and how can this be done?


As we might expect, the report raises more questions than answers. If you think a lot about this space, it is worth reading -- it proposes a classification of problems that's different from ones I've seen before -- but I really only expect it to be useful to people actively doing research in this space.

Machine Learning and Artificial Intelligence: how do we make sure technology serves the open society?

A report from a workshop held in the UK in December 2017, where a diverse group of people ("omputer scientists, entrepreneurs, business leaders, politicians, journalists, researchers and a pastor") talked about the "impact of AI on societies, governments and the relations between states, and between states and companies". I'm glad to see more thought being put into AI policy, and especially what the effects of AI may be on world stability.


If you haven't thought about digital forgery and its implications before, I strongly recommend it; otherwise I'd recommend you only read the section titled "A weekend experiment."

The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation

Commoditisation of AI, digital forgery and the end of trust: how we can fix it

Giorgio Patrini, Simone Lini, Hamish Ivey-Law and Morten Dahl.

This blog post talks about the many issues that could arise as digital forgery of audio and video gets better, and also talks about potential solutions and their pitfalls, such as digital signatures, and training ML models that can distinguish between real and fake media. He even has a simple but well-done experiment that looks at the ability of an ML model to see whether or not a face swap has happened.


‘Skynet’ Revisited: The Dangerous Allure of Nuclear Command Automation

While I won't summarize this article in full here, I found it useful to see how some academics are thinking about the risks of automation in the military, as well as to get a picture of what current automation efforts actually look like. One quote I found particularly interesting:

"You will find no stronger proponent of integration of AI capabilities writ large into the Department of Defense," said Lieutenant General Jack Shanahan, director of the Joint Artificial Intelligence Center (JAIC), at a September 2019 conference at Georgetown University, "but there is one area where I pause, and it has to do with nuclear command and control." Referring to [an] article's assertion that an automated U.S. nuclear launch ability is needed, he said, "I read that. And my immediate answer is, 'No. We do not.'"


I think the ability of the AI community to cause things to happen via activism is quite important: it seems much more likely that if AI x-risk concerns are serious, we will be able to convince the AI community of them, rather than say the government, or company executives. This mechanism of action seems much more like the "epistemic community" model used in this paper: we would be using our position as experts on AI to convince decision makers to take appropriate precautions with sufficiently powerful AI systems. Applying the discussion from the paper to this case, we get the perhaps unsurprising conclusion that it is primarily important that we build consensus amongst AI researchers about how risky any particular system is.

Activism by the AI Community: Analysing Recent Achievements and Future Prospects

The AI community has been surprisingly effective at activism: it has led to discussions of a ban on lethal autonomous weapons systems (LAWS), created several initiatives on safety and ethics, and has won several victories through organizing (e.g. Project Maven). What explains this success, and should we expect it to continue in the future? This paper looks at this through two lenses.

First, the AI community can be considered an _epistemic community_: a network of knowledge-based experts with coherent beliefs and values on a relevant topic. This seems particularly relevant for LAWS: the AI community clearly has relevant expertise to contribute, and policymakers are looking for good technical input. From this perspective, the main threats to future success are that the issues (such as LAWS) become less novel, that the area may become politicized, and that the community beliefs may become less cohesive.

Second, the AI community can be modeled as organized labor (akin to unions): since there is high demand for AI researchers, and their output is particularly important for company products, and the companies are more vulnerable to public pressure, AI researchers wield a lot of soft power when they are united. The main threat to this success is the growing pool of talent that will soon be available (given the emphasis on training experts in AI today), which will reduce the supply-demand imbalance, and may reduce how commited the AI community as a whole is to collective action.

Overall, it seems that the AI community has had good success at activism so far, but it is unclear whether it will continue in the future.


I definitely agree that near-term and long-term often seem to mean many different things, and I certainly support efforts to be more precise in our language.

While we're talking about near-term and long-term, I'll add in my own gripe: "long-term" implies that the effects will be felt only in the far future, even though many people focused on such effects are doing so because there's a significant probability of such effects being felt in only a few decades.

Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society

This paper argues that the existing near-term / long-term distinction conflates four different axes on which research could differ: the capability level of AI systems (current pattern-matching systems vs. future intelligent systems), the impacts of AI systems (impacts that are being felt now like fairness vs. ones that will be felt in the future like x-risks), certainty (things that will definitely be problems vs. risks that are more speculative) and extremity (whether to prioritize particularly extreme risks). While there are certainly correlations across these axes, they are not the same thing, and discourse would be significantly improved by disambiguating the axes. For example, both authors of the paper see their work as considering the medium-to-long-term impacts of near-to-medium-term AI capabilities.


Stories like this make me optimistic that we can actually coordinate AI researchers to take appropriate safety precautions when developing advanced AI systems, even if the economic incentives point in the other direction (and I'm not sure they do).

How a Pentagon Contract Became an Identity Crisis for Google

Scott Shane, Cade Metz and Daisuke Wakabayashi

After Google accepted a share of the contract for the Maven program run by the Defense Department, Google has been internally fractured, with many employees strongly opposing the use of AI for military applications.


I think quite a lot about points 1 and 3, which I think also apply to technical safety research, not just policy. For our research to have an impact, it is necessary that either the research or its authors have enough credibility to actually influence decision-makers. In addition, the problems we will face in the future could depend on technical work done today: for example, if we were convinced that (say) AIs trained via evolution are too risky, we could push for AI to be developed in other ways now.

AI Alignment Podcast: On the Long-term Importance of Current AI Policy

Lucas Perry, Nicolas Moës and Jared Brown

While this podcast focuses both on the details of current policy as well as the long-term impact of engaging in policy today, I'm mostly interested in the latter, and so will simply quote Lucas's summary of points for that part:

1) Experience gained on short-term AI policy issues is important to be considered a relevant advisor on long-term AI policy issues coming up in the future.
2) There are very few people that care about AGI safety currently in government, politics or in policy communities.
3) There are opportunities to influence current AI policy decisions in order to provide a fertile ground for future policy decisions or, better but rarer, to be directly shaping AGI safety policy today through evergreen texts. Future policy that is implemented is path dependent on current policy that we implement today. What we do now is precedent setting.
4) There are opportunities today to develop a skillset useful for other policy issues and causes.
5) Little resource is being spent on this avenue for impact, so the current return on investment is quite good.


FLI Podcast: Distributing the Benefits of AI via the Windfall Clause

[Last week](https://mailchi.mp/9d279b575b1a/an-88-how-the-principal-agent-literature-relates-to-ai-risk), we had a brief summary of the [Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/) paper. This podcast goes into more depth about the potential benefits and objections to this clause: it's in some sense a more accessible and conversational elaboration of many of the points made in the paper.


I enjoyed this paper; it seems good to have a better picture of the potential future of openness in AI research, for the reasons given in [Strategic Implications of Openness in AI Development](https://www.nickbostrom.com/papers/openness.pdf). You could also imagine patent pools as a vehicle for safety, as they are one possible way by which companies can cooperate to ensure a shared commitment to safety (along the lines of <@OpenAI's charter@>(@OpenAI Charter@)): they could tie competitiveness (which requires use of the research protected by the patent pool) to safety (the conditions involved in licensing the research in the patent pool).

Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter

This paper analyzes intellectual property (IP) considerations as they relate to AI. They identify two main incentives for companies: first, to publish AI research openly in order to attract top talent, and second, holding enough patents that they can credibly threaten to sue other companies for patent infringement. This second criterion lets companies stay in a mutually-assured-destruction (MAD) scenario, where if any one company litigates for patent infringement, they will quickly be met with a countersuit, and so the (fragile) equilibrium is to avoid litigation. They also identify two incentives for governments: first, to provide patents as a financial incentive for innovation in order to incentivize research, and second, to allow their own national security apparatus to use state of the art research while keeping it secret from perceived rivals.

Based on this analysis, they propose three scenarios that could unfold in the future. First, the status quo continues, in which companies keep acquiring patents in order to maintain the MAD equilibrium. Second, the equilibrium breaks, with one company litigating that then causes all the other companies to also litigate. This could result in most research becoming secret, in order to ensure that other companies can't "steal" the work and get a patent first. Similarly, contributions to open-source research might decrease, as it would be particularly easy to use such contributions as evidence of patent infringement. Third, more "patent pools" get created, in which multiple companies pool their patents together, to reduce the risk of litigation. Such patent pools could also be used to enforce other principles: with a sufficiently large patent pool, it could be the case that in order to remain competitive actors must license from the patent pool, and such licensing agreements could enforce specific ethical principles (although it would have to be careful to avoid violating antitrust law).