I enjoyed this paper; it seems good to have a better picture of the potential future of openness in AI research, for the reasons given in [Strategic Implications of Openness in AI Development](https://www.nickbostrom.com/papers/openness.pdf). You could also imagine patent pools as a vehicle for safety, as they are one possible way by which companies can cooperate to ensure a shared commitment to safety (along the lines of <@OpenAI's charter@>(@OpenAI Charter@)): they could tie competitiveness (which requires use of the research protected by the patent pool) to safety (the conditions involved in licensing the research in the patent pool).

Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter

This paper analyzes intellectual property (IP) considerations as they relate to AI. They identify two main incentives for companies: first, to publish AI research openly in order to attract top talent, and second, holding enough patents that they can credibly threaten to sue other companies for patent infringement. This second criterion lets companies stay in a mutually-assured-destruction (MAD) scenario, where if any one company litigates for patent infringement, they will quickly be met with a countersuit, and so the (fragile) equilibrium is to avoid litigation. They also identify two incentives for governments: first, to provide patents as a financial incentive for innovation in order to incentivize research, and second, to allow their own national security apparatus to use state of the art research while keeping it secret from perceived rivals.

Based on this analysis, they propose three scenarios that could unfold in the future. First, the status quo continues, in which companies keep acquiring patents in order to maintain the MAD equilibrium. Second, the equilibrium breaks, with one company litigating that then causes all the other companies to also litigate. This could result in most research becoming secret, in order to ensure that other companies can't "steal" the work and get a patent first. Similarly, contributions to open-source research might decrease, as it would be particularly easy to use such contributions as evidence of patent infringement. Third, more "patent pools" get created, in which multiple companies pool their patents together, to reduce the risk of litigation. Such patent pools could also be used to enforce other principles: with a sufficiently large patent pool, it could be the case that in order to remain competitive actors must license from the patent pool, and such licensing agreements could enforce specific ethical principles (although it would have to be careful to avoid violating antitrust law).


While data efficiency is often motivated in AI by the promise of applications where data is limited, I am actually more excited about the thresholding effects mentioned in the paper, in which as you get the last little bit of performance left to get, the ML systems become robust enough to enable applications to be built on top of them (in the way that computer vision is (hopefully) robust enough that self-driving cars can be built on top of CV models). It seems quite likely to me that data efficiency and large data collection efforts together will tend to mean that the newest ML applications will happen in large firms rather than startups due to these thresholding effects. See also [Value of the Long Tail](https://www.lesswrong.com/posts/Nbcs5Fe2cxQuzje4K/value-of-the-long-tail).

I disagree with the point about distributional shift. I often think of ML as a search for a function that does the right thing on the training data. We start with a very large set of functions, and then as we get more training data, we can rule out more and more of the functions. The problem of distribution shift is that even after this, there's a large set of functions left over, and the ML model implements an arbitrary one of them, rather than the one we wanted; while this function behaves as we want on the training set, it may not do so on the test set.

"Increased data efficiency" means that we have some way of ruling out functions a priori that doesn't require access to data, or we get better at figuring out which functions should be ruled out by the data. Suppose for concreteness that our original ML algorithm gets 90% performance with 1,000 samples, and 95% with 10,000 samples, and our efficient ML algorithm that e.g. incorporates causality gets 95% performance with 1,000 samples. Then the question we now have is "do the 9000 extra samples eliminate more bad functions than the assumption of causality, given that they both end up with 95% performance?" My guess would be that the "assumption of causality" does a better job of eliminating bad functions, because it will probably generalize outside of the training set, whereas the 9000 extra samples won't. (This doesn't depend on the fact that it is "causality" per se, just that it is a human-generated intuition.) So this suggests that the efficient ML algorithm would be _more_ robust than the original one.

Social and Governance Implications of Improved Data Efficiency

Aaron D. Tucker, Markus Anderljung, Allan Dafoe

Few-shot learning, meta learning, transfer learning, active learning: there's a lot of types of learning that are aiming to improve the data efficiency of ML techniques. What happens if we succeed? This paper propose two effects: an _access effect_, by which smaller actors can start using ML capabilities with their smaller amounts of data, and a _performance effect_, by which existing actors see improvements in the performance of their AI systems (since their existing data goes further than it used to). It then analyzes some societal implications of these effects.

By making it easier to reach a given performance with limited data, we will gain access to new applications where data is limited (e.g. machine translation of ancient languages), and for existing applications, more actors will be able to use ML capabilities (this also includes bad actors, who can more easily pursue malicious applications). However, it is not clear how this will affect the competitive advantage of large AI firms: while more actors can access a given level of performance, which might suggest more competition, the large AI firms also gain performance, which could reverse the effect. For example, improved data efficiency makes no difference in a pure winner-take-all situation, and _advantages_ the large firms in cases where the last few miles of performance lead to large gains in utility (e.g. self-driving cars).

The paper also makes two comments on the impacts for AI safety: that algorithms based on human oversight will become more competitive (as it will be more reasonable to collect expensive human data), and that distributional shift problems may become worse (since if you train on smaller amounts of data, you are less likely to see "rare" inputs).


Should Artificial Intelligence Governance be Centralised? Design Lessons from History

Peter Cihon, Matthijs Maas, and Luke Kemp

This paper tackles the question of whether or not AI governance should be centralized. In favor of centralized governance, we see that a centralized institution can have major political power, and can be more efficient by avoiding duplication of work and making it easier for actors to comply. However, a centralized institution often requires a large amount of time to create, and even afterwards, it tends to be slow-moving and so may not be able to respond to new situations easily. It also leads to a single point of failure (e.g. via regulatory capture). It also may be forced to have a relatively light touch in order to ensure buy-in from all the relevant actors. With a decentralized system, you can get _forum shopping_ in which actors select the governance mechanisms they like best, which can lead to a quicker progress on time-sensitive issues, but can also lead to weakened agreements, so it is not clear whether this is on net a good effect.

The paper then applies this framework to high-level machine intelligence (HLMI), a particular operationalization of powerful AI, and concludes that centralization is particularly promising for HLMI governance.


Lessons for Artificial Intelligence from Other Global Risks

Seth D. Baum, Robert de Neufville, Anthony M. Barrett, Gary Ackerman

This paper looks at other areas of risk (biorisk, nuclear weapons, global warming, and asteroid collision) and applies lessons from these areas to AI risk. I'd recommend reading the paper: the stories for each area are interesting but hard to summarize here.


France, China, and the EU All Have an AI Strategy. Shouldn’t the US?


2018 White House Summit on Artificial Intelligence for American Industry


I disagree with Wendell that current machine ethics will be necessary for AI alignment -- that might be the case, but it seems like things change significantly once our AI systems are smart enough to actually understand our moral systems, so that we no longer need to design special procedures to embed ethical reasoning in the AI system.

It does seem useful to have coordination on governance, along the lines of governance coordinating committees; it seems a lot better if there's only one or two groups that we need to convince of the importance of an issue, rather than 53 (!!).

AI Alignment Podcast: Machine Ethics and AI Governance

Machine ethics has aimed to figure out how to embed ethical reasoning in automated systems of today. In contrast, AI alignment starts from an assumption of intelligence, and then asks how to make the system behave well. Wendell expects that we will have to go through stages of development where we figure out how to embed moral reasoning in less intelligent systems before we can solve AI alignment.

Generally in governance, there's a problem that technologies are easy to regulate early on, but that's when we don't know what regulations would be good. Governance has become harder now, because it has become very crowded: there are more than 53 lists of principles for artificial intelligence and lots of proposed regulations and laws. One potential mitigation would be **governance coordinating committees**: a sort of issues manager that keeps track of a field, maps the issues and gaps, and figures out how they could be addressed.

In the intermediate term, the worry is that AI systems are giving increasing power to those who want to manipulate human behavior. In addition, job loss is a real issue. One possibility is that we could tax corporations relative to how many workers they laid off and how many jobs they created.

Thinking about AGI, governments should probably not be involved now (besides perhaps funding some of the research), since we have so little clarity on what the problem is and what needs to be done. We do need people monitoring risks, but there’s a pretty robust existing community doing this, so government doesn't need to be involved.


AI policy careers in the EU


While it is consistent to believe that OpenAI was just generating hype since GPT-2 was predictably not going to have major misuse applications, and this has now been borne out, I'm primarily glad that we started thinking about publication norms _before_ we had dangerous models, and it seems plausible to me that OpenAI was also thinking along these lines.

GPT-2: 1.5B Release

Irene Solaiman, Jack Clark, Miles Brundage

Along with the release of the last and biggest GPT-2 model, OpenAI explains their findings with their research in the time period that the staged release bought them. While GPT-2 can produce reasonably convincing outputs that are hard to detect and can be finetuned for e.g. generation of synthetic propaganda, so far they have not seen any evidence of actual misuse.


Special Report: AI Policy and China – Realities of State-Led Development


It seems like a lot of the best policy work is happening at OpenAI (see for example the [OpenAI charter](https://blog.openai.com/openai-charter/)), I strongly encourage people to apply!

Policy Researcher

There is a job opportunity at OpenAI as a policy researcher, which does not seem to have any formal requirements.


FLI Podcast: Beyond the Arms Race Narrative: AI and China


Information security careers for GCR reduction

This post suggests that information security could be a good career path for people looking to reduce global catastrophic risks (GCRs). For AI in particular, such experts could help mitigate attacks by malicious or incautious actors to steal AI-related intellectual property. It also reduces the risk of destabilizing AI technology races. Separately, such experts could think about the potentially transformative impact of AI on cyber offense and defense, develop or advise on credible commitment techniques (see eg. <@model governance@>(@Scaling shared model governance via model splitting@)), or apply the [security mindset](https://intelligence.org/2013/07/31/ai-risk-and-the-security-mindset/) more broadly.


Where are people thinking and talking about global coordination for AI safety?


It's interesting to see that this group went with a very similar release strategy, and I wish they had written more about why they chose to do what they did. I do like that they are on the face of it "cooperating" with OpenAI, but eventually we need norms for _how_ to make publication decisions, rather than always following the precedent set by someone prior. Though I suppose there could be a bit more risk with their models -- while they are the same size as the released GPT-2 models, they are better tuned for generating propaganda than GPT-2 is.

Grover: A State-of-the-Art Defense against Neural Fake News

Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, Yejin Choi

Could we use ML to detect fake news generated by other ML models? This paper suggests that models that are used to generate fake news will also be able to be used to _detect_ that same fake news. In particular, they train a GAN-like language model on news articles, that they dub GROVER, and show that the generated articles are _better_ propaganda than those generated by humans, but they can at least be detected by GROVER itself.

Notably, they do plan to release their models, so that other researchers can also work on the problem of detecting fake news. They are following a similar release strategy as with <@GPT-2@>(@Better Language Models and Their Implications@): they are making the 117M and 345M parameter models public, and releasing their 1.5B parameter model to researchers who sign a release form.


I broadly agree with this article -- I can't imagine how a council that meets four times a year could properly provide advice on Google's AI projects. I'm not sure if the solution is more powerful and intensive ethics councils whose primary power is public accountability. I expect that making good decisions about AI ethics requires either a technical background, or a long, detailed conversation with a person with that background, neither of which are possible with the public. This could mean that an ethics board could struggle to raise a legitimate issue, or that they could cause outrage about an issue that is upon closer examination not an issue at all. I would feel better about a board with some more formal power, such as the ability to create investigations that could lead to fines, the ability to sue Google, specific whistleblowing affordances, etc. (I have no idea how feasible any of those suggestions are, even assuming Google was okay with them.)

On the tweetstorm about OpenAI, I'm not sure if I've said it before in this newsletter, but I generally trust OpenAI to be trying to do the right thing, and this is one of the reasons for that. Of course, I also know and trust many people who work there.

Google’s brand-new AI ethics board is already falling apart

Google [announced](https://www.blog.google/technology/ai/external-advisory-council-help-advance-responsible-development-ai/) an ethical advisory council, that quickly became controversial, and was then [cancelled](https://www.vox.com/future-perfect/2019/4/4/18295933/google-cancels-ai-ethics-board). The author makes the point that the council was not well-placed to actually advise on ethics -- it would only meet four times a year, and could only give recommendations. This committee, and others at Facebook and Microsoft, seem to be more about PR and less about AI ethics. Instead, an AI ethics council should include both insiders and outsiders, should be able to make formal, specific, detailed recommendations, and would publicly announce whether the recommendations were followed. **Key quote:** "The brouhaha has convinced me that Google needs an AI ethics board quite badly — but not the kind it seems to want to try to build."

In a [tweetstorm](https://twitter.com/KelseyTuoc/status/1113544870625308673), the author holds OpenAI up as a large organization that is at least *trying* to engage deeply with AI ethics, as evidenced by their safety and policy team, their <@charter@>(@OpenAI Charter@), and <@GPT-2@>(@Better Language Models and Their Implications@). They make public, contentful statements that are weird, controversial and seem bad from a PR perspective. The arguments they make and hear about AI ethics and policy lead to real decisions with consequences.


Rationally Speaking #231 - Helen Toner on "Misconceptions about China and artificial intelligence"

In this podcast Helen talks about AI policy, China, and the Center for Security and Emerging Technology, where she is the director of strategy. Some of her opinions that stood out to me:

    - While Baidu is a huge tech company and is the main search engine, it's a bit misleading to call it the Google of China, since it doesn't have the same diversity of products that Google does.
    - While the social credit score story seems overblown, the reporting on the Uighur situation seems to be basically accurate.
    - Based on a very small sample of AI researchers in China, it seems like Chinese researchers are less interested in thinking about the real-world effects of the technology they're building, relative to Western researchers.
    - Since people in government have so little time to think about so many issues, they have simple versions of important ideas. For example, it's easy to conclude that China must have an intrinsic advantage at data since they have more people and fewer privacy controls. However, there's a lot of nuance: for example, the entire Internet is in English, which seems like a big advantage for the US.
    - The incentives in China can be quite different: in at least one case, a professor's salary depended on the number of papers published.
    - A particularly interesting question: "how does it help the US geopolitically if an American company is developing powerful AI?"


FLI Podcast: Why Ban Lethal Autonomous Weapons?

Ariel Conn, Emilia Javorsky, Bonnie Docherty, Ray Acheson, and Rasha Abdul Rahim


A Survey of the EU's AI Ecosystem

This report analyzes the European AI ecosystem. The key advantage that Europe has is a strong focus on ethical AI, as opposed to the US and China that are more focused on capabilities research. However, Europe does face a significant challenge in staying competitive with AI, as it lacks both startup/VC funding as well as talented researchers (who are often going to other countries). While there are initiatives meant to help with this problem, it is too early to tell whether they will have an impact. The report also recommends having large multinational projects, along the lines of CERN and the Human Brain Project. See also [Import AI](https://jack-clark.net/2019/03/25/making-better-healthcare-ai-systems-via-audio-de-identification-teaching-drones-to-help-humans-fight-fires-and-why-language-models-could-be-smarter-than-you-think/).


I noticed a variant of this effect myself while reading GPT-2 results -- my brain very quickly fell into the mode of skimming without absorbing anything, though it felt more like I had made the evaluation that there was nothing to gain from the content, which seems okay if the goal is to avoid fake news. I also find this to be particularly interesting evidence about the differences between our low-level, effortless pattern matching, as well as our more effortful and accurate "logical reasoning".

Humans Who Are Not Concentrating Are Not General Intelligences

This post argues that humans who skim the stories produced by [GPT-2](https://blog.openai.com/better-language-models/) ([AN #46](https://mailchi.mp/c48f996a5db5/alignment-newsletter-46)) would not be able to tell that they were generated by a machine, because while skimming we are not able to notice the obvious logical inconsistencies in its writing. Key quote: "OpenAI HAS achieved the ability to pass the Turing test against humans on autopilot". This suggests that fake news, social manipulation, etc. will become much easier. However, it might also force people to learn the skill of detecting the difference between humans and bots, which could let them learn to tell when they are actively focusing on something and are "actually learning" as opposed to skimming for "low order correlations".


It's pretty strange to me that different governments would take such different approaches to AI - this suggests that either academics, think tanks, policy analysts etc. do not agree on the risks, or that there isn't enough political pressure for some of the risks to make it into the strategies. It seems like the AI community would have a significant opportunity to shape policy in the latter case -- I'd imagine for example that an open letter signed by thousands of researchers could be quite helpful in creating political will. (Of course, creating a comprehensive open letter that most researchers will approve of might be quite hard to do.)

Toward AI Security: Global Aspirations for a More Resilient Future

This report analyzes various AI security risks (including both near-term and long-term concerns) and categorizes them, and then analyzes how different national strategies and policies have engaged with these risks. Most interestingly (to me) it comes to the conclusion that most national AI strategies are focused on very different areas and often ignore (in the sense of not mentioning) risks that other countries have highlighted, though there are still some areas for cooperation, such as improving the transparency and accountability of AI systems.


Risk factors for s-risks

This post discusses four risk factors for creating extreme disvalue in the universe (s-risks): advanced technology, lack of effort to avoid those outcomes, inadequate security and law enforcement, and polarisation and divergence of values. Tobias notes that he's most worried about cases where most of these factors occur, because the absence of any of them mitigates the threat posed by the others.


Ethical and societal implications of algorithms, data, and artificial intelligence: a roadmap for research

Nuffield Foundation and Leverhulme Centre for the Future of Intelligence


Summary of the 2018 Department of Defense Artificial Intelligence Strategy


Bridging near- and long-term concerns about AI


FLI Podcast- Artificial Intelligence: American Attitudes and Trends

This is a podcast about [The American Public’s Attitudes Concerning Artificial Intelligence](https://www.fhi.ox.ac.uk/aipublic2019/) ([AN #41](https://mailchi.mp/8c3f02cabccd/alignment-newsletter-41)), you can see my very brief summary of that.


The American Public’s Attitudes Concerning Artificial Intelligence

This presents results from a survey of Americans about their attitudes towards AI. There's not a compelling objective story I can tell, so you might as well look at the executive summary, which presents a few interesting highlights. One interesting fact: the median person thinks that we're more likely than not to have "high-level machine intelligence" within 10 years! You could also read [Vox's take](https://www.vox.com/future-perfect/2019/1/9/18174081/fhi-govai-ai-safety-american-public-worried-ai-catastrophe), which emphasizes that the public is concerned about long-term AI risk.


Perspective in Frontiers in Computer Science

Robust Artificial Intelligence and Robust Human Organizations


Game Changers: AI Part III, AI and Public Policy


On the (In)Applicability of Corporate Rights Cases to Digital Minds


China’s Grand AI Ambitions with Jeff Ding


I'm glad to see this systematic exploration of an issue that the AI safety community has consistently had to grapple with. I would have liked to see a more nuanced definition of misinformation than "information that is already clearly false", since it's not always obvious what qualifies as clearly false, and since there are many varieties of misinformation.

Superintelligence Skepticism as a Political Tool

Countering Superintelligence Misinformation

Two ways to have better discussions about superintelligence are correcting misconceptions, and preventing misinformation from being spread in the first place. The latter might be achieved by educating prominent voices, creating reputational costs to misinformers (both individuals and companies), focusing media attention, etc. Research suggests the former is very difficult; strategies include addressing pre-existing motivations for believing misinformation and using advance warnings to 'inoculate' people against false claims.


The role of corporations in addressing AI’s ethical dilemmas


Podcast: Artificial Intelligence – Global Governance, National Policy, and Public Trust with Allan Dafoe and Jessica Cussins

Allan Dafoe, Jessica Cussins, and Ariel Conn

Topics discussed include the difference between AI governance and AI policy, externalities and solving them through regulation, whether governments and bureaucracies can keep up with AI research, the extent to which the US' policy of not regulating AI may cause citizens to lose trust, labor displacement and inequality, and AI races.


Superintelligence Skepticism as a Political Tool


The Advent of Huang's Law


It's a short, interesting piece, and it's got some actual numbers and quotes from Xu Li (one of the co-founders of the startup, SenseTime), so you should read it.

China Now Has the Most Valuable AI Startup in the World


The Deep Roots and Long Branches of Chinese Technonationalism


The Facets of Artificial Intelligence: A Framework to Track the Evolution of AI

Fernando Martınez-Plumed, Bao Sheng Loe, Peter Flach, Sean ́Oh ́Eigeartaigh, Karina Vold and Jose Hernandez-Orallo


The four claims seem quite plausible to me. The post also argues that this suggests that the AI community should create its own third-party institution rather than depending on a state-led institution, but I didn’t really follow the argument for this, nor do I agree with the conclusion. On one hand, it’s plausible that the AI community could create such an institution before some crisis, while states could not (claim 2), and that such a community-led institution would be more binding on researchers across different countries (part of claim 3). But on the other hand, such institutions seem much worse at binding companies (from which I expect most of the risk) and presumably would have much less context than a state-led institution (claim 1).

What the AI Community Can Learn From Sneezing Ferrets and a Mutant Virus Debate

If you can modify bird flu to be transmitted in ferrets, should your experimental methods be published in full? When this question arose, the National Science Advisory Board for Biosecurity (NSABB) unanimously recommended that key methodological details should not be published. The World Health Organization (WHO) disagreed, calling for full publication in order to enable better science, and arguing that it would be too hard to create a mechanism to grant researchers with a legitimate need access to the redacted information. At this point, many bird flu researchers declared a voluntary moratorium on such research, until the controversy settled. Ultimately, the NSABB reversed its position and the paper was published.

This post suggests four lessons for the AI community to learn:

1. **Third-party institutions like the NSABB can lead to better-considered outcomes.** In particular, they can counteract publish-or-perish incentives and provide additional expertise and context (the NSABB had clearance for secret information that researchers could not access).
2. **These institutions don’t happen “by default”.** The NSABB was only established after the anthrax attacks of 2001, and most other countries don’t have an analogous body.
3. **However, the powers of such institutions are limited.** The NSABB is geographically limited and was not able to create a mechanism for sharing information to only those with legitimate need.
4. **Researchers must take on some responsibility as well.** For example, the voluntary moratorium allowed for the development of better policy.


AI Policy Challenges and Recommendations


Collective Action on Artificial Intelligence: A Primer and Review

This paper reviews much of the work in AI governance (specifically, work on AI races and other collective action problems).


AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries

Peter Cihon, Moritz J. Kleinaltenkamp, Jonas Schuett, Seth D. Baum

_Certification_ is a method of reducing information asymmetries: it presents credible information about a product to an audience that they couldn’t have easily gotten otherwise. With AI systems, certification could be used to credibly share information between AI actors, which could promote trust amongst competitors, or to share safety measures to prevent a race to the bottom on safety, caused by worrying that “the other guys would be even more unsafe”. Certification is at its best when there is _demand_ from an audience to see such certificates; public education about the need for credible information can help generate such demand.

However, certification often runs into problems. _Symbol-substance decoupling_ happens when certificates are issued to systems that don’t meet the standards for certification. For example, in “ethics washing”, companies advertise a self-certificate in which their products are approved by ethics boards, but those ethics boards have no real power. _Means-ends decoupling_ happens when the standards for certification don’t advance the goals for which the certificate was designed. For example, a certificate might focus on whether a system was tested, rather than on what test was conducted, leading applicants to use easy-to-pass tests that don’t actually provide a check on whether the method is safe.

Effective certification for future AI systems needs to be responsive to changes in AI technology. This can be achieved in a few ways: first, we can try to test the underlying goals which are more likely to remain stable; for example, we could certify ethical principles that will likely remain the same in the future. Second, we can match the certification to the types of people and institutions, that is, our certifications talk about the executives, citizens, or corporations (rather than e.g. specific algorithms that may be replaced in the future). Third, the certification system can build in mechanisms for updating the certification criteria periodically.

The paper then analyzes seven existing certification systems for AI systems; you’ll have to read the paper for details.


Case studies of self-governance to reduce technology risk

Should we expect AI companies to reduce risk through self-governance? This post investigates six historical cases, of which the two most successful were the Asilomar conference on recombinant DNA and the actions of Leo Szilard and other physicists in 1939 (around the development of the atomic bomb). It is hard to make any confident conclusions, but the author identifies the following five factors that make self-governance more likely:

1. The risks are salient.
2. If self-governance doesn’t happen, then the government will step in with regulation (which is expected to be poorly designed).
3. The field is small, so that coordination is easier.
4. There is support from gatekeepers (e.g. academic journals).
5. There is support from credentialed scientists.


Some AI Governance Research Ideas

Exactly what it says.


State of AI


While some of the discussed measures like marking autonomous weapon systems are very specific to the military use of AI, others such as the measures focussed on T&E could be useful more broadly to reduce risks from competitive pressures around AI. I believe that military AI use is the largest source of AI risk in the next few years, so I am glad that people are working on this.

AI and International Stability: Risks and Confidence-Building Measures

Militaries are likely incentivized to integrate machine learning in their operations and because AI is a general-purpose technology, we cannot expect militaries to not use it at all. Still, it matters a great deal how and for which purposes militaries use AI. While militaries are currently not spending a lot on AI, there are several risks from broader adoption: An acceleration of warfare, and ensuing pressure for more automation as well as increased difficulty of managing escalation.  More difficulties in assessing others' strength and less immediate human cost of conflict, leading to more risk-taking. Accidents due to AI systems' brittleness being mistaken as attacks and inflaming tensions.

This paper explores confidence-building measures (CBMs) as a way to reduce the negative effects of military AI use on international stability. CBMs were an important tool during the Cold War. However, as CBMs rely on a shared interest to succeed, their adoption has proven challenging in the context of cybersecurity, where the stakes of conflict are less clear than in the Cold War. The authors present a set of CBMs that could diminish risks from military use of AI and discuss their advantages and downsides. On the broad side, these include building norms around the military use of AI, dialogues between civil actors with expertise in the military use of AI from different countries, military to military dialogues, and code of conducts with multilateral support. On the more specific side, states could engage in public signalling of the importance of Test and Evaluation (T&E), transparency about T&E processes and push for international standards for military AI T&E. In addition, they could cooperate on civilian AI safety research, agree on specific rules to prevent accidental escalation (similar to the Incidents at Sea Agreement from the Cold War), clearly mark autonomous systems as such, and declare certain areas as off-limits for autonomous systems. Regarding nuclear weapons, the authors suggest an agreement between states to retain exclusive human control over nuclear launch decisions and a prohibition of uninhabited nuclear launch platforms such as submarines or bombers armed with nuclear weapons.


I am optimistic about developing AGI collaboratively, especially through AI researchers cooperating. I'm not sure whether external incentives from government are the right way to achieve this -- it seems likely that such regulation would be aimed at the wrong problems if it originated from government and not from AI researchers themselves. I'm more optimistic about some AI researchers developing guidelines and incentive structures themselves, that researchers buy into voluntarily, that maybe later get codified into law by governments, or adopted by companies for their AI research.

Shaping economic incentives for collaborative AGI

This post considers how to encourage a culture of cooperation among AI researchers. Then, when researchers try to create AGI, this culture of cooperation may make it more likely that AGI is developed collaboratively, instead of with race dynamics, making it more likely to be safe. It specifically poses the question of what external economic or policy incentives could encourage such cooperation.


Reading through this, it seems that countries are taking quite different approaches towards AI. I don't know what to make of this -- are they acting close to optimally given their geopolitical situation (which must then vary a lot by country), or does no one know what's going on and as a result all of the strategies are somewhat randomly chosen? (Here by "randomly chosen" I mean that the strategies that one group of analysts would select with is only weakly correlated with the strategies another group would select.) It could also be that the approaches are not actually that different.

An Overview of National AI Strategies

A short reference on the AI policies released by various countries.


I’m certainly on board with the goal of working together towards shared goals. That being said, I don't fully understand what's being proposed here: how exactly is an incompletely theorized agreement supposed to be made? Is this more of a “shared ethos” that gets spread by word of mouth, or is there a document that people sign on to? If there is a document, what goes into it, who would agree to it, and how binding is it? I’d be excited to see more work fleshing out these concrete details, or even better, actually causing such an agreement to exist in practice.

Bridging the Gap: The Case for an ‘Incompletely Theorized Agreement’ on AI Policy.

Like <@several@>(@Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society@) <@past@>(@Bridging near- and long-term concerns about AI@) <@papers@>(@Medium-Term Artificial Intelligence and Society@), this paper argues that the differences between the “near-term” and “long-term” communities are probably exaggerated. Collaboration between these communities would be particularly beneficial, since it could prevent the field of AI policy from becoming fragmented and ineffectual, which is especially important now while the field is nascent and there is political will for AI policy progress.

The authors propose the notion of an “incompletely theorized agreement” in order to foster this sort of collaboration. In an incompletely theorized agreement, the parties agree to suspend disagreement on some thorny theoretical question, in order to coordinate action towards a shared pragmatic purpose. Such agreements could be used to set aside relatively unimportant disagreements between the two communities, in favor of pursuing goals that both communities care about. For example, we could imagine that such an agreement would allow both communities to push for more and better reflection by AI researchers on the impacts of the systems that they build, or to enable action that ensures we preserve the integrity of public discourse and informed decision-making (e.g. by regulating AI-enabled disinformation).


Fragmentation and the Future: Investigating Architectures for International AI Governance

Peter Cihon, Matthijs M. Maas, Luke Kemp

Should AI governance be done centrally, through an international body, or in a fragmented, decentralized fashion? This paper identifies various considerations pointing in different directions:

1. Centralized institutions can have more political power when designed well: their regulations can have more “teeth”.
2. Centralized institutions can be more efficient from the participant’s perspectives: if there is only one set of regulations, it is much easier for each participant to adhere to those regulations.
3. A centralized institution will typically be slower to act, as there are many more parties with a larger stake in the outcome. This can make it brittle, especially when the pace of technological change outpaces that of regulatory change.
4. Centralized institutions face a breadth vs. depth dilemma: if the regulations are too stringent, then some actors (i.e. nations, companies, etc) won’t participate (there is depth but not breadth), and similarly, to get everyone to participate the regulations must often be quite weak (breadth but not depth). In contrast, with decentralized approaches, the depth of the regulations can be customized to each participant.
5. With more fragmented approaches, actors can “forum shop” for the regulations which they think are best. It is unclear whether this is helpful or harmful for AI governance.
6. It is unclear which approach leads to more coordination. While a centralized approach ensures that everyone has the same policies, leading to policy _coherence_, it does not necessarily mean that those policies are good. A decentralized approach could lead to faster adaptation leading to better policies that are then copied by others, leading to more effective coordination overall.


Our AI governance grantmaking so far

This post describes Open Philanthropy’s approach to AI governance, in which they focus on governance for worlds in which we have transformative AI (that is, AI which is as impactful as the Industrial Revolution). However, there is a major challenge: it is usually quite unclear whether a proposed intermediate goal is even positive. Something like “increased safety regulation in the US and Europe” might initially seem good if done well, but even if done well it may actually _increase_ risk by (say) privileging AI development in countries that have lower standards than what would have existed without the regulations. Given this effect, it’s hard to do [hits-based giving](https://www.openphilanthropy.org/blog/hits-based-giving): most intermediate goals have relatively low expected value, because the huge positive value scenarios can be canceled out by the huge negative value scenarios, and so not many things look like a “hit”.

As a result, grantmaking has so far focused on intermediate goals that seem robustly neutral-to-good: some kinds of research (including on which intermediate goals would be robustly good), advocacy for intermediate goals that are robustly good (e.g. methods for gaining high assurance in AI systems), broad field-building, and better-informed AI governance training and advice.

The post also has some examples of AI governance work that the author thinks have been useful.


Future Indices: How Crowd Forecasting Can Inform the Big Picture

Michael Page, Catherine Aiken, Dewey Murdick

This paper explains the methodology behind CSET’s recent forecasting project, Foretell. We would like to know which of several potential geopolitical scenarios might happen in the next 3-7 years. We can get some insight into this by asking relevant experts for their opinions, but often many experts will disagree, making it hard to know what to conclude.

We’d like to mitigate this by leveraging the wisdom of the crowds. Unfortunately, this would require us to have a clear and precise operationalization of our scenarios; the scenarios we’re interested in are rarely amenable to such operationalization. Instead, we can find a number of _predictors_ that would argue for a specific scenario, and identify one or more _metrics_ which are themselves clear and precise and give us information about some predictor. We can get forecasts for these metrics using the wisdom of the crowds. We can then compute the deviations between crowd forecasts and simple trend extrapolations of historical data, and use the observed trend directions as arguments for or against particular scenarios.

The paper illustrates this in the case of potential scenarios involving the US, China, and AI. An example of an important predictor is “US-China tensions”. Associated metrics include the amount of US-China trade, the number of Chinese O visas, etc. In this case, the crowd predictions suggested trend deviations in the metrics that argued for increasing US-China tensions.


AI Nationalism

As AI becomes more important in the coming years, there will be an increasing amount of "AI nationalism". AI policy will be extremely important and governments will compete on keeping AI talent. For example, they are likely to start blocking company takeovers and acquisitions that cross national borders -- for example, the UK could have been in a much stronger position had they blocked the acquisition of DeepMind (which is UK-based) by Google (which is US-based).


Decision Points in AI Governance

While the last couple of years have seen a proliferation of “principles” for the implementation of AI systems in the real world, we are only now getting to the stage in which we turn these principles into practice. During this period, _decision points_ are concrete actions taken by some AI stakeholder with the goal of shaping the development and use of AI. (These actions should not be predetermined by existing law and practice.) Decision points are the actions that will have a disproportionately large influence on the field, and thus are important to analyze. This paper analyzes three case studies of decision points, and draws lessons for future decision points.

First, we have the Microsoft AETHER committee. Like many other companies, Microsoft has established a committee to help the company make responsible choices about its use of AI. Unlike e.g. [Google’s AI ethics board](https://www.vox.com/future-perfect/2019/4/4/18295933/google-cancels-ai-ethics-board), this committee has actually had an impact on Microsoft’s decisions, and has published several papers on AI governance along the way. The committee attributes its success in part to executive-level support, regular opportunities for employee and expert engagement, and integration with the company’s legal team.

Second, we have the <@GPT-2@>(@Better Language Models and Their Implications@) staged release process. We’ve <@covered@>(@The Hacker Learns to Trust@) <@this@>(@When Is It Appropriate to Publish High-Stakes AI Research?@) <@before@>(@Grover: A State-of-the-Art Defense against Neural Fake News@), so I won’t retell the story here. However, this shows how a deviation from the norm (of always publishing) can lead to a large discussion about what publication norms are actually appropriate, leading to large changes in the field as a whole.

Finally, we have the OECD AI Policy Observatory, a resource that has been established to help countries implement the OECD AI principles. The author emphasizes that it was quite impressive for the AI principles to even get the support that they did, given the rhetoric about countries competing on AI. Now, as the AI principles have to be put into practice, the observatory provides several resources for countries that should help in ensuring that implementation actually happens.


Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and Governance

Seán S. ÓhÉigeartaigh, Jess Whittlestone, Yang Liu, Yi Zeng, Zhe Liu

This paper argues that it is important that AI ethics and governance is cross-cultural, and provides a few recommendations towards this goal:
1. Develop AI ethics and governance research agendas requiring cross-cultural cooperation
2. Translate key papers and reports
3. Alternate continents for major AI research conferences and ethics and governance conferences
4. Establish joint and/or exchange programmes for PhD students and postdocs


Accounting for the Neglected Dimensions of AI Progress

Fernando Martinez-Plumed, Shahar Avin, Miles Brundage, Allan Dafoe, Sean O


Antitrust-Compliant AI Industry Self-Regulation

One way to reduce the risk of unsafe AI systems is to have agreements between corporations that promote risk reduction measures. However, such agreements may run afoul of antitrust laws. This paper suggests that this sort of self-regulation could be done under the “Rule of Reason”, in which a learned profession (such as “AI engineering”) may self-regulate in order to correct a market failure, as long as the effects of such a regulation promote rather than harm competition.

In the case of AI, if AI engineers self-regulate, this could be argued as correcting the information asymmetry between the AI engineers (who know about risks) and the users of the AI system (who don’t). In addition, since AI engineers arguably do not have a monetary incentive, the self-regulation need not be anticompetitive. Thus, this seems like a plausible method by which AI self-regulation could occur without running afoul of antitrust law, and so is worthy of more investigation.


I found this line particularly interesting: "We will design our AI systems to be appropriately cautious, and seek to develop them in accordance with best practices in AI safety research." It sounds like the time is ripe for someone to write a "best practices" paper!

AI at Google: our principles

Following the outcry over the Maven program, Google has written a blog post detailing the principles they will follow for AI.


I’ve been pretty skeptical of the requirement to write a broader impacts statement. My experience of it was primarily one of frustration, for a few reasons:
1. Forecasting the future is hard. I don’t expect a shallow effort to forecast to be all that correlated with the truth. There were lots of simple things I could say that “sound” right but that I don’t particularly expect to be true, such as “improving cooperation in multiagent RL will help build cooperative, helpful personal assistants”. It’s a lot harder to say things that are actually true; a real attempt to do this would typically be a paper in itself.
2. To the extent that the statement does affect reviews, I expect that reviewers want to hear the simple things that sound right; and if I _don’t_ write them, it would probably be a strike against the paper.
3. Even if I did write a good statement, I don’t expect anyone to read it or care about it.

From a birds-eye view, I was also worried that if such statements do become popular, they’ll tend to ossify and build consensus around fairly shallow views that people come up with after just a bit of thought.

I do think many of the proposals in this paper would help quite a bit, and there probably is a version of these statements that I would like and endorse.

Institutionalizing ethics in AI through broader impact requirements

Carina E. A. Prunkl, Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, Allan Dafoe

This short perspective analyzes the policy implemented by NeurIPS last year in which paper submissions were required to have a section discussing the broader impacts of the research. Potential benefits include _anticipating_ potential impacts of research, _acting_ to improve these impacts, _reflecting_ on what research to do given the potential impacts, and improving _coordination_ across the community. However, the policy may also lead to _trivialization_ of ethics and governance (thinking that all the relevant thinking about impacts can be done in this single statement), _negative attitudes_ towards the burden of writing such statements or responsible research in general, a _false sense of security_ that the ethics are being handled, and a _perception_ of ethics as something to be done as an afterthought.

The main challenges that can cause these sorts of negative effects are:
1. Analyzing broader impacts can be difficult and complex,
2. There are not yet any best practices or guidance,
3. There isn’t a clear explanation of the purpose of the statements, or transparency into how they will be evaluated,
4. It’s tempting to focus on the research that determines whether or not your paper is published, rather than the broader impacts statement which mostly does not affect decisions,
5. Researchers may have incentives to emphasize the beneficial impacts of their work and downplay the negative impacts.
6. Biases like motivated reasoning may affect the quality and comprehensiveness of impact statements.

To mitigate these challenges, the authors recommend improving _transparency_, setting _expectations_, providing _guidance_ on how to write statements, improving _incentives_ for creating good impact statements, and learning from experience through _community deliberation_. To improve incentives in particular, broader impact statements could be made an explicit part of peer review which can affect acceptance decisions. These reviews could be improved by involving experts in ethics and governance. Prizes could also be given for outstanding impact statements, similarly to best paper awards.


AI Alignment Podcast: On Lethal Autonomous Weapons

Paul Scharre, author of "Army of None: Autonomous Weapons and the Future of War", talks about various issues around Lethal Autonomous Weapons (LAWs), including the difficulty to talk about an arms race around autonomous weapons when different people mean different things by "arms race" and autonomy comes in varying degrees, the military's need for reliability in the context of AI systems' lack of robustness to distributional shift and adversarial attacks, whether the law of war correctly deals with LAWs, as well as the merits and problems of having a human in the loop.

While autonomous weapons are unlikely to directly contribute to existential risk, efforts to establish limits on them could be valuable by creating networks and preparing institutions for collaboration and cooperation around future AI issues.


It makes sense that role playing can help find extreme, edge case scenarios. I'm not sure how likely I should find such scenarios -- are they plausible but unlikely (because forecasting is hard but not impossible), or are they implausible (because it would be very hard to model an _entire government_, and no one person is going to do it justice)? Note that according to the paper, the prior literature on role playing is quite positive (though of course it's talking about role playing in other contexts, e.g. business and military contexts). Still, this seems like quite an important question that strongly impacts how seriously I take the results of these role playing scenarios.

Exploring AI Futures Through Role Play

Shahar Avin, Ross Gruetzemacher, James Fox

This paper argues that role playing (akin to the "wargames" used in the military) is a good way to explore possible AI futures, especially to discover unusual edge cases, in a 10-30 year time horizon. Each player is assigned a role (e.g. director of AI at Tencent, or president of the US) and asked to play out their role faithfully. Each game turn covers 2 simulated years, in which players can negotiate and take public and private actions. The game facilitator determines what happens in the simulated world based on these actions. While early games were unstructured, recent games have had an AI "tech tree", that determines what AI applications can be developed.

From the games played so far, the authors have found a few patterns:
- Cooperation between actors on AI safety and (some) restriction on destabilizing uses of AI seem to both be robustly beneficial.
- Even when earlier advances are risky, or when current advances are of unclear value, players tend to pursue AI R&D quite strongly.
- Many possible kinds of coalitions are possible, e.g. between governments, between corporations, between governments and corporations, and between sub-roles within a corporation.


AI Algorithms Need FDA-Style Drug Trials

Olaf J. Groth, Mark J. Nitzberg, Stuart J. Russell


An Interview with Ben Garfinkel

Joshua Monrad, Mojmír Stehlík and Ben Garfinkel

AI seems poised to be a very big deal, possibly through the development of AGI, and it's very hard to forecast what would happen next. However, looking at history, we can see a few very large trajectory shifts, such as the Agricultural Revolution and Industrial Revolution, where everything changed radically. We shouldn't assume that such change must be for the better. Even though it's hard to predict what will happen, we can still do work that seems robustly good regardless of the specific long-term risk. For example, Ben is optimistic about research into avoiding adversarial dynamics between different groups invested in AI, research into how groups can make credible commitments, and better forecasting. However, credible commitments are probably less tractable for AI than with nukes or biological weapons because AI systems don't leave a large physical footprint, can easily proliferate, and are not a clear category that can be easily defined.


I quite strongly agree that the most important impact of the GPT-2 decision was that it has started a discussion about what appropriate safety norms should be, whereas before there were no such norms at all. I don't know whether or not GPT-2 is dangerous, but I am glad that AI researchers have started thinking about whether and how publication norms should change.

The Hacker Learns to Trust

An independent researcher attempted to replicate <@GPT-2@>(@Better Language Models and Their Implications@) and was planning to release the model. However, he has now decided not to release, because releasing would set a bad precedent. Regardless of whether or not GPT-2 is dangerous, at some point in the future, we will develop AI systems that really are dangerous, and we need to have adequate norms then that allow researchers to take their time and evaluate the potential issues and then make an informed decision about what to do. **Key quote:** "sending a message that it is ok, even celebrated, for a lone individual to unilaterally go against reasonable safety concerns of other researchers is not a good message to send".


I definitely agree that having everyone follow the same review norms is important: it doesn't do much good to hold back from publishing something problematic if a different group will publish all of the details a few weeks later. However, getting everyone to agree on a change to the existing norms seems incredibly hard to do, though it might be feasible if it was limited to only the largest actors who can engage deeply in the debate of what these norms should be.

When Is It Appropriate to Publish High-Stakes AI Research?

Claire Leibowicz, Steven Adler, and Peter Eckersley

Following the <@GPT-2 controversy@>(@Better Language Models and Their Implications@), the Partnership on AI held a dinner with OpenAI and other members of the AI community to discuss the tension between the norm of openness and the desire to mitigate potential unintended consequences and misuse risks of AI research. The post discusses some of the relevant considerations, and highlights a key conclusion: while there is not yet a consensus on on review norms for AI research, there *is* a consensus that **whatever the review norms are, they should be standardized across the AI community**.


Global AI Talent Report 2019

This report has a lot of statistics on the growth of the field of AI over the last year.


Advice to UN High-level Panel on Digital Cooperation

Luke Kemp, Peter Cihon,Matthijs Michiel Maas, Haydn Belfield,Dr Seán Ó hÉigeartaigh, Jade Leung, and Zoe Cremer


Surveying Safety-relevant AI Characteristics

Jose Hernandez-Orallo, Fernando Martınez-Plumed, Shahar Avin, Seán Ó hÉigeartaigh


Since this report discusses neither its methodology nor the implications of its findings, it's difficult to know what conclusions to draw from it. The overall priorities seem to be roughly what I would have expected, except that I'm positively surprised by how much emphasis was placed on ethics.

Building an AI World

This report summarises the AI strategies released by 18 different countries and regions. In particular, it rates how much emphasis they put on each of 8 areas. Broadly speaking, countries were most focused on Industrial strategy, Research, and AI talent (in that order), moderately focused on Ethics and Data, and least focused on Future of work, AI in governments, and Inclusion.


2018 IEEE Global Communications Conference (GLOBECOM)

In the particular setting that they're considering, let's say that we require C compute to train M. Then one of the organizations had to contribute at least 0.5C, and that organization could defect by investing 1.5C. The first 0.5C is used to take part in the model splitting scheme so as not to arouse suspicion, and the remaining 1C is used to train a new version of M from scratch. So, security in this setting requires us to assume that the organization is unwilling to invest 3x the compute they are going to invest. This assumption seems questionable, but when it does hold, the evidence from the paper suggests that model splitting is relatively secure, since it typically takes an additional 0.4-1C in order to fully solve the model completion problem.

When there are N >> 2 parties, each party only has to contribute C/N. So, the assumption that no party will use C compute to recreate the model now translates to an assumption that no party will invest (N+1)C compute, which seems more reasonable for sufficiently large N.

I suspect we can get more mileage if each party had its own training data that it kept secret. It's not clear how to train an AI system such that the training data remains secret, but if we could do that, and the model was split across each group, it would probably be impossible for any one group to recover a new model M' that achieved performance as good as that of M.

Scaling shared model governance via model splitting

Miljan Martic, Jan Leike, Andrew Trask, Matteo Hessel, Shane Legg, Pushmeet Kohli

Suppose that two organizations want to develop a deep learning model together without allowing either one to unilaterally use the model. This can be done cryptographically using homomorphic encryption or secure multiparty computation, but this introduces several orders of magnitude of slowdown. What about the much simpler solution of letting each organization have half of the parameters, that are not shared with the other organization? For this to be secure, it should be prohibitively difficult to find the other organization's parameters. In the least convenient world where each organization has access to all training data, hyperparameters etc., this is the security of the _model completion problem_, where given all of the normal setup for deep learning as well as half of the trained parameters for a model M, the goal is to create a new model that performs as well as M. Of course, we can simply rerun the training procedure that was used to create M, so the cost is bounded above by the cost to create M in the first place. We might be able to do better by leveraging the trained parameters that we know -- for example, by using those parameters as an initialization for the model instead of whatever initialization we normally use. The paper empirically investigates how well strategies like this can work. They find that it is relatively easy to create a model that achieves good performance (getting 80% of the way to the best performance), but quite difficult to achieve performance as good as that of M, typically requiring 40-100% of the time it took to create M.


FLI Signs Safe Face Pledge


Handful of Countries – Including the US and Russia – Hamper Discussions to Ban Killer Robots at UN


This may be useful for AI policy researchers, since it explores which hardware is being made by which companies in which locations, and some of the geopolitical implications. While it's a long paper, AI researchers could probably skip the first half without missing much.

Computational Power and the Social Impact of Artificial Intelligence

This paper contains a lot of introductory material about hardware in ML, plus details on where it's being made. It discusses the competition between China and the US to dominate hardware production. Hwang notes that the trend towards more specialised hardware may decrease the price of implementing ML but also decrease flexibility after deployment. He points out that simulation learning, self-play and meta-learning reduce the need for data at the expense of increased compute, which may increase hardware's importance going forward.


How Will National Security Considerations Affect Antitrust Decisions in AI? An Examination of Historical Precedents

This paper looks at whether historically the US has used antitrust law to advance unrelated national security objectives, and concludes that it is rare and especially recently economic considerations tend to be given more weight than national security considerations.


Medium-Term Artificial Intelligence and Society

Like a <@previously summarized paper@>(@Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society@), this paper aims to find common ground between near-term and long-term priorities in medium-term concerns. This can be defined along several dimensions of an AI system: when it chronologically appears, how feasible it is to build it, how certain it is that we can build it, how capable the system is, how impactful the system is, and how urgent it is to work on it.

The paper formulates and evaluates the plausibility of the _medium term AI hypothesis_: that there is an intermediate time period in which AI technology and accompanying societal issues are important from both presentist and futurist perspectives. However, it does not come to a strong opinion on whether the hypothesis is true or not.


Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning


How Europe might matter for AI governance


To stay ahead of Chinese AI, senators want new commission


AI in the UK: ready, willing and able?


EU Member States sign up to cooperate on Artificial Intelligence


Podcast: Six Experts Explain the Killer Robots Debate

Paul Scharre, Toby Walsh, Richard Moyes, Mary Wareham, Bonnie Docherty, Peter Asaro, and Ariel Conn


Narrow AI Nanny: Reaching Strategic Advantage via Narrow AI to Prevent Creation of the Dangerous Superintelligence


Joint Artificial Intelligence Center Created Under DoD CIO


Artificial Intelligence and International Affairs: Disruption Anticipated


India's National Strategy for Artificial Intelligence


Tianjin city in China eyes US$16 billion fund for AI work, dwarfing EU’s plan to spend US$1.78 billion


Initial Reference Architecture of an Intelligent Autonomous Agent for Cyber Defense

Alexander Kott, Luigi V Mancini, Paul Théron, Martin Drašar, Edlira Dushku, Heiko Günther, Markus Kont, Benoît LeBlanc, Agostino Panico, Mauno Pihelgas, Krzysztof Rzadca

See [Import AI's summary](https://jack-clark.net/2018/04/02/importai-88-nato-designs-a-cyber-defense-ai-object-detection-improves-with-yolov3-france-unveils-its-national-ai-strategy/).


France's AI strategy


This is a more accessible introduction to the weirdness of the universal prior than the [original post](https://ordinaryideas.wordpress.com/2016/11/30/what-does-the-universal-prior-actually-look-like/), but I think it is missing a lot of details that were present, so if you're confused by some aspect, it may be worth checking out the original post.

What does the universal prior actually look like?

Clarifying Consequentialists in the Solomonoff Prior

In the universal Solomonoff prior, since there is no time bound on how long the Turing machines can run for, some short Turing machines could encode universes in which life develops, figures out that it can influence us through the prior, and starts to predict strings that we care about but changes them slightly to influence our decisions. These could be much shorter than the intended "natural" Turing machine that "correctly" predicts the string.


Anthropics confuses me and I haven't prioritized understanding it yet, so I'm going to abstain.

[Probability is fake, frequency is real](https://www.lesswrong.com/posts/vmfW2qTac4vF3YS3J/probability-is-fake-frequency-is-real) and [Repeated (and improved) Sleeping Beauty problem](https://www.lesswrong.com/posts/KZh9eKCkBRbevkGnP/repeated-and-improved-sleeping-beauty-problem)

Attacks the Sleeping Beauty problem in anthropics.


I think this is anthropomorphizing the AI too much. To the extent that a (current) reinforcement learning system can be said to "have goals", the goal is to maximize reward, so wireheading actually is furthering its current goal. It might be that in the future the systems we design are more analogous to humans and then such an approach might be useful.

A framework for thinking about wireheading

Humans don't wirehead (take heroin, which gives huge positive reward) because it does not further their current goals. Maybe analogously we could design an AI that realizes that wireheading would not help it achieve its current goals and so wouldn't wirehead.


The post, and a [comment](https://www.lesswrong.com/posts/DfewqowdzDdCD7S9y/agents-that-learn-from-human-behavior-can-t-learn-human#EeG2tLbzc3W2yuLX7), note that the difference between Alice and Bob is that if Alice received further information (from a moral philosopher, maybe), she'd start maximizing a specific one of the utility functions. The value learning agent could notice this and correctly infer the utility function. It could also actively propose the information to Alice and see how she responds.

Agents That Learn From Human Behavior Can't Learn Human Values That Humans Haven't Learned Yet

Suppose Alice has moral uncertainty over five utility functions, and so optimizes a weighted combination of them; while Bob's true utility function is the same weighted combination of the utility functions. Alice and Bob will mostly act the same, and so a value learning agent wouldn't be able to distinguish between them.


I'm confused (I'm not an expert in this field), but I'm not sure what I'm confused about. Is there a dynamics model? Given that the agent gets access to a prior, can it find Pr(U | o, a) and choose the a with maximum expected utility? Why are we including reflection? There are often many fixed points, which one do we pick?

An environment for studying counterfactuals

Proposes a class of environments in which the agent is tasked with predicting the utility of every action, in addition to maximizing expected utility. It is evaluated on the utility achieved as well as correctly predicting the utility it gets. Epsilon-exploration is required, so for every action there is always some chance that the agent will be tested on predicting the utility of that action. The agent is also provided a prior P about the world, including what the agent will do (which exists due to a fixed-point theorem).


This shows that, for a particular class of ways that the reward function might be wrong, the corresponding policy is still only slightly suboptimal. This doesn't make me feel much better about having an incorrect reward function, as it feels like this class of ways is a small subset of the ways in which we'll be wrong in practice. I do think the identification of what has to go wrong for really bad outcomes is useful, and I'd be interested to see experiments with the proposed reward learning algorithm.

Bounding Goodhart's Law

Derives a bound on the regret from having a misspecified reward function. Essentially the regret comes from two main sources -- cases where the misspecified reward assigns too high a reward to states that the misspecified policy visits a lot, and cases where the misspecified reward assigns too low a probability to states that the true best policy would visit a lot. The post also proposes an algorithm for reward learning that takes into account these insights.


I think that it is correct to optimize for your current utility function, and I didn't find the argument for the other side convincing (and wrote a comment on the post with more details).

Self-Modification of Policy and Utility Function in Rational Agents

Choosing to Choose?

If it is possible for your utility function to change, then should you optimize for your current utility function, or your expected future utility function? The post gives an argument for both sides, and ultimately says that you should optimize for your current utility function, but notes some problems with the proposed argument for it.


I agree that this is an important idea, and it has arisen elsewhere -- in ML, this is part of the thinking on the problem of generalization. There are no-free-lunch theorems that say you cannot do well in arbitrary environments, where the constructions typically violate the "strong correlation between different experiences" heuristic. In philosophy, this is the problem of induction.

Conditioning, Counterfactuals, Exploration, and Gears

One way that you can think about counterfactuals is to _condition_ on some low probability state, and then look at the probability distribution that implies. This seems like the most general version of counterfactuals, but it doesn't match what we intuitively mean by counterfactuals, which is more like "suppose that by fiat this constraint were met, but don't consider what would have caused it, now predict the consequences". This sort of imputing only works because there are very simple rules governing our universe, so that there are strong correlations between different experiences and so it actually is possible to generalize to very new situations. It seems very important to use this idea in order to advance beyond epsilon-exploration for new situations.


I do think that classifiers with a bounded task that run for a bounded amount of time are unlikely to develop unaligned subagents with memory. However, I still feel very unclear on the term "unaligned subagent", so I'm not very confident in this assessment.

Conditions under which misaligned subagents can (not) arise in classifiers

Agents or subagents with "goals" are only likely to arise when you are considering tasks where it is important to keep state/memory, because past inputs are informative about future inputs. So, unaligned subagents are unlikely to arise for eg. classification tasks where it is not necessary to model how things change over time.


Decision-theoretic problems and Theories; An (Incomplete) comparative list

It's just what it says in the title -- a list of problems in decision theory, and what particular decision theories recommend for those problems.


I found the part about how people prove things to be the most interesting part of the post, because my own method seems different from both. I usually alternate between searching for solutions, counterexamples to solutions, and proving that solutions must satisfy some property.

Dependent Type Theory and Zero-Shot Reasoning

Humans can do zero-shot reasoning (in the sense of writing down proofs) by "running a type checker in their head" (analogous to theorem provers like Lean). The post gives an example of this, using Lean syntax. However, humans seem to have very different ways of thinking -- for example, you could either generate ideas for solutions to a problem, see if they work, and iterate, or you could start proving some facts about the problem, and keep on proving things until you have proved a solution. These feel like many-shot reasoning and zero-shot reasoning respectively, even though they are both attempting a zero-shot task. This is one way to understand the difference between Iterated distillation and amplification, and Agent foundations -- the former is many-shot and the latter is zero-shot, even though both are attempting a zero-shot task.


This isn't my area of expertise, but it seems right. It feels very weird to claim that having more knowledge makes you worse off in general, but doesn't seem impossible.

Logical Uncertainty and Functional Decision Theory

If an agent has logical uncertainty about what action it will take, then the agent seems more likely to reason about counterfactuals correctly. For example, this would likely solve the [5-and-10 problem](https://agentfoundations.org/item?id=1399). Without logical uncertainty, an agent that knows about itself can be one of many different fixed points, many of which can be quite bad.


I'm all for having better definitions that make things clearer and easier to reason about. I don't know if "ease of proofs" is the right thing to aim for -- "ease of reasoning" is closer to what I care about, even if it's informal reasoning.

Mathematical Mindset

Introduces a new term, "mathematical mindset", which is about finding good _definitions_ or _models_ that make it easier for you to reason about them. For example, you expect proofs with a newer definition to be shorter or more general. Key quote: "Having a “mathematical mindset” means being comfortable with words being redefined. This is because it means being comfortable with models being upgraded -- in particular, with models being related and compared to each other: the activity of theorization."


I don't know where a proposed definition is supposed to be used so it's hard for me to comment on how relevant this objection is.

Monk Treehouse: some problems defining simulation

Some approaches to AI alignment require you to identify copies of programs in the environment, and it is not clear how to do this in full generality. Proposals so far have attempted to define two programs to be equivalent if they do the same thing now and would also do the same thing in counterfactual worlds. This post argues that such definitions don't work using an analogy where there are monks computing by moving heavy stones in a treehouse, that could unbalance it. In this setting, there are lots of checks and balances to make sure that the program does one and only one thing; any counterfactual you specify would lead to weird results (like the treehouse falling over from unbalanced stones, or monks noticing that something is off and correcting the result, etc.) and so it wouldn't be considered equivalent to the same program on a silicon-based computer.


Non-exploitability is my preferred resolution to Pascal's mugging. However, it seems like such an obvious solution, yet there's very little discussion of it, which makes me think that there's some fatal flaw that I'm not seeing.

No, I won't go there, it feels like you're trying to Pascal-mug me

One explanation for why [Pascal's mugging](https://en.wikipedia.org/wiki/Pascal%27s_mugging) feels intuitively wrong is that if we were to pay the mugger, we would open ourselves up to exploitation by any other agent. [Logical induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/) puts uncertainties on statements in such a way that it isn't exploitable by polynomial-time traders. Perhaps there is a connection here that can help us create AIs that don't get mugged.


Seems like a reasonable starting point to measure optimization power. As Alex Mennen [notes](https://www.lesswrong.com/posts/vWfFDahpnY3tnCLrp/a-universal-score-for-optimizers#e3Hsaqkc2xuWsHY2t), it's dependent on the specific action set chosen, and doesn't take into account the strength of preferences, only their ranking.

A universal score for optimizers

We can measure the optimization power of an agent as the log probability that a random agent matches the outcome that the agent achieves.


Unfortunately, the "desirable properties" for these problems all seem to conspire to make any particular problem fairly low impact.

Alignment problems for economists

What AI alignment problems could we outsource to economists? There are some who would be interested in working on alignment, but don't because it would be too much of a career risk.


In any formally defined game where the agent has perfect information (including about other agents), utility functions do in fact determine what an agent should do -- but in many cases, this seems to go against our intuitions (as in the Ultimatum game, for example). I don't think that the way to resolve this is to introduce more values; I think it is that the maximization step in maximizing expected utility depends a lot on the environment you're in, and any formalization is going to miss out on some important aspects of the real world, leading to different answers. (For example, in a repeated Ultimatum game, I would expect fairness to arise naturally.)

Conceptual problems with utility functions

It's strange to use utility functions to model agents, because often utility functions do not determine the outcome in games with multiple agents, such as the Ultimatum game, and we have to resolve the situation with "meta-values" like fairness. So, instead of using utility functions, we need to have a conception of agency where the "values" are part of the decisionmaking process.


I think I've summarizes this post badly, because I didn't really understand it. In particular, I didn't understand the jump from "humans do approximate inference over the space of models" to "counterfactuals form the initialization".

On the Role of Counterfactuals in Learning

This post hypothesizes that since humans are computationally bounded, we infer causal models using approximate inference (eg. Gibbs sampling), as opposed to a full Bayesian update. However, approximate inference algorithms depend a lot on choosing a good initialization. Counterfactuals fill this role.


The motivation was to use this to deduce the intentions of a superintelligent AI system, but it seems that such an AI system could figure out it is being tested and respond in the "expected" way.

The Intentional Agency Experiment

In order to determine whether an agent has some intention, we can check to see whether the agent would take actions that achieve the intent under a wide range of circumstances (either counterfactuals, or actual changes to the environment). For example, to show that an ant has agency and intends to find sugar, we could block its route to the sugar and notice that it finds a path around the obstacle.


I'm somewhat confused -- I'm not sure what the point is here.

Two agents can have the same source code and optimise different utility functions

Even if you have two agents with identical source code, their goals are in relation to themselves, so each agent will, for example, try to gain resources for itself. Since the two agents are now competing, they clearly have different utility functions.


The approach here is to learn heuristics that can guide a top-level search algorithm, which is the sort of thing that I think deep learning is particularly well poised to improve right now. Note that gradient descent is a top-level search algorithm here, because a separate loss function is constructed _for every protein_, rather than having a single loss function that is used to train a network that works on all proteins. However, unlike other applications such as SMT solvers, the top-level search algorithm does not have some sort of "correctness" guarantee.

AlphaFold: Using AI for scientific discovery

Andrew Senior, John Jumper and Demis Hassabis

This post briefly describes AlphaFold, a system that does well at the protein folding problem. They train neural networks that can be used to evaluate how good a particular proposed protein structure is. This can be used to guide an evolutionary search that repeatedly replaces pieces of a protein structure with new protein fragments from a generative model. Alternatively, it can be used to construct a loss function for entire proteins, which allows us to use gradient descent to optimize the protein structure.


Neural-Augmented Static Analysis of Android Communication

Jinman Zhao, Aws Albarghouthi, Vaibhav Rastogi, Somesh Jha, Damien Octeau


LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations

Michael Schaarschmidt, Alexander Kuhnle, Ben Ellis, Kai Fricke, Felix Gessert, Eiko Yoneki


As interpretability goes, using a modular model with human-interpretable intermediate representations seems quite good -- it decouples the problem of understanding the model's output into two smaller problems. The big downside is that it requires a lot more labeling (877 segmented images in this case), and that the human-interpretable representation may not be the best one for the job. For example, if there are other visual cues besides the specific features DeepMind used that help with recommending treatment, this model will not be able to take advantage of them, while an end-to-end trained system could.

A major milestone for the treatment of eye disease

DeepMind's partnership with Moorfields Eye Hospital has resulted in an AI system that can recognize features of eye disease and recommend treatment. Interestingly, in order to get interpretability, they train two networks instead of one, where one predicts the features of eye disease for all of the tissue (eg. haemorrhages, lesions and irregular fluids), and the other then makes a recommendation for treatment. This required them to label a subset of the dataset with feature markers in order to train the first model.


I take this as an example of what advanced AI systems will look like -- a system of different modules, each with its own job, passing around information appropriately in order to perform some broad task. Some of the modules could be neural nets (which can learn hard-to-program functions), while others could be classic programs (which generalize much better and are more efficient). OpenAI Five also has elements of this -- the drafting system is a classic program operating on the win probabilities from the neural net. It's also interesting how many tricks are required to get Smart Linkify to work -- I don't know whether to think that this means generally intelligent AI is further away, or that the generally intelligent AI that we build will rely on these sorts of tricks.

The Machine Learning Behind Android Smart Linkify

Android now has Smart Linkify technology, which allows it to automatically find pieces of text that should link to another app (for example, addresses should link to Maps, dates and times to Calendar, etc). There are a lot of interesting details on what had to be done to get this to actually work in the real world. The system has two separate nets -- one which generates candidate entities, and another which says what kind of entity each one is. In between these two nets, we have a regular program that takes the set of proposed entities, and prunes it so that no two entities overlap, and then sends it off to the entity classification net. There are a few tricks to get the memory requirements down, and many dataset augmentation tricks to get the nets to learn particular rules that it would not otherwise have learned.


Tackling Climate Change with Machine Learning

David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, Alexandra Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Kording, Carla Gomes, Andrew Y. Ng, Demis Hassabis, John C. Platt, Felix Creutzig, Jennifer Chayes, Yoshua Bengio

See [Import AI](https://jack-clark.net/2019/06/17/import-ai-151-us-army-trains-starcraft-ii-ai-teaching-drones-to-dodge-thrown-objects-and-fighting-climate-change-with-machine-learning/).


This is a great introduction and overview for people who want to better understand the gaps between current RL and practically deployable RL. I do wish the authors had spent more time explaining and clarifying the design of their proposed testbed system, since the descriptions of it are all fairly high level.

Challenges of Real-World Reinforcement Learning

Gabriel Dulac-Arnold, Daniel Mankowitz, Todd Hester

This paper is a fairly clear and well-done literature review focusing on the difficulties that will need to be overcome in order to train and deploy reinforcement learning on real-world problems. They describe each of these challenges - which range from slow simulation speeds, to the need to frequently learn off-policy, to the importance of safety in real world systems - and for each propose or refer to an existing metric to capture how well a given RL model addresses the challenge. Finally, they propose a modified version of a humanoid environment with some of these real-world-style challenges baked in, and encourage other researchers to test systems within this framework.


Applying Deep Learning To Airbnb Search


Learning Scheduling Algorithms for Data Processing Clusters

Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, Mohammad Alizadeh


Predicted Variables in Programming

Victor Carbune, Thierry Coppey, Alexander Daryin, Thomas Deselaers, Nikhil Sarda, Jay Yagnik


Evidence against current methods leading to human level artificial intelligence

This post briefly lists arguments that current AI techniques will not lead to high-level machine intelligence (HLMI), without taking a stance on how strong these arguments are.


There certainly is a lot of hype both around putting human intelligence in silicon, as well as the risks that surround such an endeavor. Even though I focus on such risks, I agree with Jordan that these are overhyped in the media and we would benefit from having more faithful coverage of them. I do disagree on some specific points. For example, he says that human-imitative AI is not sufficient to build some AI systems such as self-driving cars, but why couldn't an AI with human intelligence just do whatever humans would do to build self-driving cars? (I can think of answers, such as "we don't know how to give the AI system access to all the data that humans have access to", but I wish he had engaged more with this argument.) I do agree with the overall conclusion that in the near future humans will make progress on building such systems, and not by trying to give the systems "human intelligence". I also suspect that we disagree either on how close we are to human-imitative AI, or at what point it is worth it to start thinking about the associated risks, but it's hard to tell more from the article.

Artificial Intelligence — The Revolution Hasn’t Happened Yet

There is a lot of hype at the moment around AI, particularly around creating AI systems that have human intelligence, since the thrill (and fear) of creating human intelligence in silicon causes overexuberance and excessive media attention. However, we _actually_ want to create AI systems that can help us improve our lives, often by doing things that humans are not capable of. In order to accomplish this, it is likely better to work directly on these problems, since human-like intelligence is neither necessary nor sufficient to build such systems. However, as with all new technologies, there are associated challenges and opportunities with these AI systems, and we are currently at risk of not seeing these because we are too focused on human intelligence in particular.


This is worth reading for the sake of knowing what deep learning cannot currently do, which I think the articles are correct about. Two of the pieces also predict that these flaws will continue in the future, but none of the arguments are very compelling to me -- I think the jury is out on that question, I could see it going either way. (This is based not on this post, but on my memories of three of the four linked pieces, which I had read before starting this newsletter.)

Deep learning - deeper flaws?

This post summarizes four pieces about the flaws of deep learning. I don't think they'll benefit from more summarization on my part.


A narrowing of AI research?

Joel Klinger, Juan Mateos-Garcia, Konstantinos Stathoulopoulos

Technology development can often be _path-dependent_, where initial poorly-thought-out design choices can persist even after they are recognized as poorly thought out. For example, the QWERTY keyboard persists to this day, because once enough typists had learned to use it, there was too high a cost to switch over to a better-designed keyboard. This suggests that we want to maintain a diversity of approaches to AI so that we can choose amongst the best options, rather than getting locked into a suboptimal approach early on.

The paper then argues, based on an analysis of arXiv papers, that thematic diversity in AI has been going down over time, as more and more papers are focused on deep learning. Thus, we may want to have policies that encourage more diversity. It also has a lot of additional analysis of the arXiv dataset for those interested in a big-picture overview of what is happening in the entire field of AI.


This is fairly superficial, if you've read any of the other things that Pearl himself has written about deep learning, you'll know all of this already.

To Build Truly Intelligent Machines, Teach Them Cause and Effect

An interview with Judea Pearl about causality, deep learning, and where the field is going.


The 30-Year Cycle In The AI Debate


This podcast covers a really impressive breadth of the existing argumentation. A lot of the reasoning is similar to <@that I’ve heard from other researchers@>(@Takeaways from safety by default interviews@). I’m really glad that Ben and others are spending time critiquing these arguments; in addition to showing us where we’re wrong, it helps us steer towards more plausible risky scenarios.

I largely agree with Ben’s criticisms of the Bostrom AI model; I think mesa-optimization is the best current case for AI risk and am excited to see more work on it. The parts of the podcast where I most disagreed with Ben were:
- I think even in the absence of solid argumentation, I feel good about a prior where AI has a non-trivial chance of being existentially threatening, partially because I think it’s reasonable to put AI in the reference class of ‘new intelligent species’ in addition to ‘new technology’.
- I’m not sure that institutions will address failures sufficiently, <@even if progress is gradual and there are warnings@>(@Possible takeaways from the coronavirus pandemic for slow AI takeoff@).

**Rohin's opinion:** I recommend listening to the full podcast, as it contains a lot of detail that wouldn't fit in this summary. Overall I agree pretty strongly with Ben. I do think that some of the counterarguments are coming from a different frame than the classic arguments. For example, a lot of the counterarguments involve an attempt to generalize from current ML practice to make claims about future AI systems. However, I usually imagine that the classic arguments are basically ignoring current ML, and instead claiming that if an AI system is superintelligent, then it must be goal-directed and have convergent instrumental subgoals. If current ML systems don't lead to goal-directed behavior, I expect that proponents of the classic arguments would say that they also won't lead to superintelligent AI systems. I'm not particularly sold on this intuition either, but I can see its appeal.

Ben Garfinkel on scrutinising classic AI risk arguments

In this podcast, Ben Garfinkel goes through several reasons why he is skeptical of classic AI risk arguments (some previously discussed <@here@>(@How Sure are we about this AI Stuff?@)). The podcast has considerably more detail and nuance than this summary.

Ben thinks that historically, it has been hard to affect transformative technologies in a way that was foreseeably good for the long-term-- it's hard e.g. to see what you could have done around the development of agriculture or industrialization that would have an impact on the world today. He thinks some potential avenues for long-term influence could be through addressing increased political instability or the possibility of lock-in, though he thinks that it’s unclear what we could do today to influence the outcome of a lock-in, especially if it’s far away.

In terms of alignment, Ben focuses on the standard set of arguments outlined in Nick Bostrom’s Superintelligence, because they are broadly influential and relatively fleshed out. Ben has several objections to these arguments:
- He thinks it isn't likely that there will be a sudden jump to extremely powerful and dangerous AI systems, and he thinks we have a much better chance of correcting problems as they come up if capabilities grow gradually.
- He thinks that making AI systems capable and making AI systems have the right goals are likely to go together.
- He thinks that just because there are many ways to create a system that behaves destructively doesn't mean that the engineering process creating that system is likely to be attracted to those destructive systems; it seems like we are unlikely to accidentally create systems that are destructive enough to end humanity.

Ben also spends a little time discussing <@mesa-optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), a much newer argument for AI risk. He largely thinks that the case for mesa-optimization hasn’t yet been fleshed out sufficiently. He also thinks it’s plausible that learning incorrect goals may be a result of having systems that are insufficiently sophisticated to represent goals appropriately. With sufficient training, we may in fact converge to the system we want.

Given the current state of argumentation, Ben thinks that it's worth EA time to flesh out newer arguments around AI risk, but also thinks that EAs who don't have a comparative advantage in AI-related topics shouldn't necessarily switch into AI. Ben thinks it's a moral outrage that we have spent less money on AI safety and governance than the 2017 movie 'The Boss Baby', starring Alec Baldwin.


The only part of this review I disagree with is the title -- every sentence in the text seems quite reasonable. I in fact do not want policy that advocates for particular solutions now, precisely because it's not yet clear what the problem actually is. (More "field-building" type policy, such as increased investment in research, seems fine.)

The review never actually argues for its title -- you need some additional argument, such as "and therefore, we will never achieve superintelligence", or "and since superintelligent AI will be like humans, they will be aligned by default". For the first one, while I could believe that we'll never build ruthlessly goal-pursuing agents for the reasons outlined in the article, I'd be shocked if we couldn't build agents that were more intelligent than us. For the second one, I agree with the outside view argument presented in _Human Compatible_: while humans might be aligned with each other (debatable, but for now let's accept it), humans are certainly not aligned with gorillas. We don't have a strong reason to say that our situation with superintelligent AI will be different from the gorillas' situation with us. (Obviously, we get to design AI systems, while gorillas didn't design us, but this is only useful if we actually have an argument why our design for AI systems will avoid the gorilla problem, and so far we don't have such an argument.)

We Shouldn’t be Scared by ‘Superintelligent A.I.’

This review of <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) argues that people worried about superintelligent AI are making a mistake by assuming that an AI system "could surpass the generality and flexibility of human intelligence while seamlessly retaining the speed, precision and programmability of a computer". It seems likely that human intelligence is strongly integrated, such that our emotions, desires, sense of autonomy, etc. are all _necessary_ for intelligence, and so general intelligence can't be separated from so-called "irrational" biases. Since we know so little about what intelligence actually looks like, we don't yet have enough information to create AI policy for the real world.


Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More


Conversation with Ernie Davis