Finite Factored Sets

This is the edited transcript of a talk introducing finite factored sets. For most readers, it will probably be the best starting point for learning about factored sets.
Video:

(Lightly edited) slides: https://intelligence.org/files/Factored-Set-Slides.pdf

(Part 1, Title Slides)   ·   ·   ·   Finite Factored Sets

(Part 1, Motivation)   ·   ·   ·   Some Context

Scott: So I want to start with some context. For people who are not already familiar with my work:

My main motivation is to reduce existential risk.
I try to do this by trying to figure out how to align advanced artificial intelligence.
I try to do this by trying to become less confused about intelligence and optimization and agency and various things in that cluster.
My main strategy here is to develop a theory of agents that are embedded in the environment that they're optimizing. I think there are a lot of open hard problems around doing this.
This leads me to do a bunch of weird math and philosophy. This talk is going to be an example of some weird math and philosophy.

For people who are already familiar with my work, I just want to say that according to my personal aesthetics, the subject of this talk is about as exciting as Logical Induction, which is to say I'm really excited about it. And I'm really excited about this audience; I'm excited to give this talk right now.

(Part 1, Table of Contents)   ·   ·   ·   Factoring the Talk

This talk can be split into 2 parts:
Part 1, a short pure-math combinatorics talk.
Part 2, a more applied and philosophical main talk.
This talk can also be split into 5 parts differentiated by color: Title Slides, Motivation, Table of Contents, Main Body, and Examples. Combining these gives us 10 parts (some of which are not contiguous):

Part 1: Short Talk
Part 2: The Main Talk

Title Slides
Finite Factored Sets
The Main Talk (It's About Time)

Motivation
Some Context
The Pearlian Paradigm

Table of Contents
Factoring the Talk
We Can Do Better

Main Body
Set Partitions, etc.
Time and Orthogonality, etc.

Examples
Enumerating Factorizations
Game of Life, etc.

(Part 1, Main Body)   ·   ·   ·   Set Partitions

All right. Here's some background math:

A partition of a set \(S\) is a set \(X\) of non-empty subsets of \(S\), called parts, such that for each \(s∈S\) there exists a unique part in \(X\) that contains \(s\).
Basically, a partition of \(S\) is a way to view \(S\) as a disjoint union. We have parts that are disjoint from each other, and they union together to form \(S\).
We'll write \(\mathrm{Part}(S)\) for the set of all partitions of \(S\).
We'll say that a partition \(X\) is trivial if it has exactly one part.
We'll use bracket notation, \([s]_{X}\), to denote the unique part in \(X\) containing \(s\). So this is like the equivalence class of a given element.
And we'll use the notation \(s∼_{X}t\) to say that two elements \(s\) and \(t\) are in the same part in \(X\).

You can also think of partitions as being like variables on your set \(S\). Viewed in that way, the values of a partition \(X\) correspond to which part an element is in.
Or you can think of \(X\) as a question that you could ask about a generic element of \(S\). If I have an element of \(S\) and it's hidden from you and you want to ask a question about it, each possible question corresponds to a partition that splits up \(S\) according to the different possible answers.
We're also going to use the lattice structure of partitions:

We'll say that \(X \geq_S Y\) (\(X\) is finer than \(Y\), and \(Y\) is coarser than \(X\)) if \(X\) makes all of the distinctions that \(Y\) makes (and possibly some more distinctions), i.e., if for all \(s,t \in S\), \(s \sim_X t\) implies \(s \sim_Y t\). You can break your set \(S\) into parts, \(Y\), and then break it into smaller parts, \(X\).
\(X\vee_S Y\) (the common refinement of \(X\) and \(Y\) ) is the coarsest partition that is finer than both \(X\) and \(Y\) . This is the unique partition that makes all of the distinctions that either \(X\) or \(Y\) makes, and no other distinctions. This is well-defined, which I'm not going to show here.

Hopefully this is mostly background. Now I want to show something new.

(Part 1, Main Body)   ·   ·   ·   Set Factorizations

A factorization of a set \(S\) is a set \(B\) of nontrivial partitions of \(S\), called factors, such that for each way of choosing one part from each factor in \(B\), there exists a unique element of \(S\) in the intersection of those parts.
So this is maybe a little bit dense. My short tagline of this is: "A factorization of \(S\) is a way to view \(S\) as a product, in the exact same way that a partition was a way to view \(S\) as a disjoint union."
If you take one definition away from this first talk, it should be the definition of factorization. I'll try to explain it from a bunch of different angles to help communicate the concept.
If \(B=\{b_0,\dots,b_{n}\}\) is a factorization of \(S\) , then there exists a bijection between \(S\) and \(b_0\times\dots\times b_{n}\) given by \(s\mapsto([s]_{b_0},\dots,[s]_{b_{n}})\). This bijection comes from sending an element of \(S\) to the tuple consisting only of parts containing that element. And as a consequence of this bijection, \(|S|=\prod_{b\in B} |b|\).
So we're really viewing \(S\) as a product of these individual factors, with no additional structure.
Although we won't prove this here, something else you can verify about factorizations is that all of the parts in a factor have to be of the same size.
We'll write \(\mathrm{Fact}(S)\) for the set of all factorizations of \(S\), and we'll say that a finite factored set is a pair \((S,B)\), where \(S\) is a finite set and \(B \in \mathrm{Fact}(S)\).
Note that the relationship between \(S\) and \(B\) is somewhat loopy. If I want to define a factored set, there are two strategies I could use. I could first introduce the \(S\), and break it into factors. Alternatively, I could first introduce the \(B\). Any time I have a finite collection of finite sets \(B\), I can take their product and thereby produce an \(S\), modulo the degenerate case where some of the sets are empty. So \(S\) can just be the product of a finite collection of arbitrary finite sets.
To my eye, this notion of factorization is extremely natural. It's basically the multiplicative analog of a set partition. And I really want to push that point, so here's another attempt to push that point:

A partition is a set \(X\) of non-empty
subsets of \(S\) such that the obvious
function from the disjoint union of
the elements of \(X\) to \(S\) is a bijection.
A factorization is a set \(B\) of non-trivial
partitions of \(S\) such that the obvious
function to the product of
the elements of \(B\) from \(S\) is a bijection.

I can take a slightly modified version of the partition definition from before and dualize a whole bunch of the words, and get out the set factorization definition.
Hopefully you're now kind of convinced that this is an extremely natural notion.

Andrew Critch: Scott, in one sense, you're treating "subset" as dual to partition, which I think is valid. And then in another sense, you're treating "factorization" as dual to partition. Those are both valid, but maybe it's worth talking about the two kinds of duality.
Scott: Yeah. I think what's going on there is that there are two ways to view a partition. You can view a partition as "that which is dual to a subset," and you can also view a partition as something that is built up out of subsets. These two different views do different things when you dualize.
Ramana Kumar: I was just going to check: You said you can start with an arbitrary \(B\) and then build the \(S\) from it. It can be literally any set, and then there's always an \(S\)…
Scott: If none of them are empty, yes, you could just take a collection of sets that are kind of arbitrary elements. And you can take their product, and you can identify with each of the elements of a set the subset of the product that projects on to that element.
Ramana Kumar: Ah. So the \(S\) in that case will just be tuples.
Scott: That's right.
Brendan Fong: Scott, given a set, I find it very easy to come up with partitions. But I find it less easy to come up with factorizations. Do you have any tricks for…?
Scott: For that, I should probably just go on to the examples.
Joseph Hirsh: Can I ask one more thing before you do that? You allow factors to have one element in them?
Scott: I said "nontrivial," which means it does not have one element.
Joseph Hirsh: "Nontrivial" means "not have one element, and not have no elements"?
Scott: No, the empty set has a partition (with no parts), and I will call that nontrivial. But the empty set thing is not that critical.

I'm now going to move on to some examples.

(Part 1, Examples)   ·   ·   ·   Enumerating Factorizations

Exercise! What are the factorizations of the set \(\{0,1,2,3\}\) ?
Spoiler space:
.
.
.
.
.
.
.
.
.
.
.
.
.
.
First, we're going to have a kind of trivial factorization:
\(\begin{split} \{ \ \ \{ \{0\},\{1\},\{2\},\{3\} \} \ \ \} \end{split} \begin{split} \ \ \ \ \underline{\ 0 \ \ \  1 \ \ \ 2 \ \ \ 3 \ } \end{split}\)
We only have one factor, and that factor is the discrete partition. You can do this for any set, as long as your set has at least two elements.
Recall that in the definition of factorization, we wanted that for each way of choosing one part from each factor, we had a unique element in the intersection of those parts. Since we only have one factor here, satisfying the definition just requires that for each way of choosing one part from the discrete partition, there exists a unique element that is in that part.
And then we want some less trivial factorizations. In order to have a factorization, we're going to need some partitions. And the product of the cardinalities of our partitions are going to have to equal the cardinality of our set \(S\) , which is 4.
The only way to express 4 as a nontrivial product is to express it as \(2 \times 2\) . Thus we're looking for factorizations that have 2 factors, where each factor has 2 parts.
We noted earlier that all of the parts in a factor have to be of the same size. So we're looking for 2 partitions that each break our 4-element set into 2 sets of size 2.
So if I'm going to have a factorization of \(\{0,1,2,3\}\) that isn't this trivial one, I'm going to have to pick 2 partitions of my 4-element set that each break the set into 2 parts of size 2. And there are 3 partitions of a 4-element sets that break it up into 2 parts of size 2. For each way of choosing a pair of these 3 partitions, I'm going to get a factorization.
\(\begin{split} \begin{Bmatrix} \ \ \ \{\{0,1\}, \{2,3\}\}, \ \\ \{\{0,2\}, \{1,3\}\} \end{Bmatrix} \end{split} \begin{split} \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \begin{array} { |c|c|c|c| } \hline 0 & 1 \\ \hline 2 & 3 \\ \hline \end{array} \end{split}\)
\(\begin{split} \begin{Bmatrix} \ \ \ \{\{0,1\}, \{2,3\}\}, \ \\ \{\{0,3\}, \{1,2\}\} \end{Bmatrix} \end{split} \begin{split} \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \begin{array} { |c|c|c|c| } \hline 0 & 1 \\ \hline 3 & 2 \\ \hline \end{array} \end{split}\)
\(\begin{split} \begin{Bmatrix} \ \ \ \{\{0,2\}, \{1,3\}\}, \ \\ \{\{0,3\}, \{1,2\}\} \end{Bmatrix} \end{split} \begin{split} \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \begin{array} { |c|c|c|c| } \hline 0 & 2 \\ \hline 3 & 1 \\ \hline \end{array} \end{split}\)
So there will be 4 factorizations of a 4-element set.
In general you can ask, "How many factorizations are there of a finite set of size \(n\) ?". Here's a little chart showing the answer for \(n \leq 25\):

\(|S|\)
\(|\mathrm{Fact}(S)|\)

0
1

1
1

2
1

3
1

4
4

5
1

6
61

7
1

8
1681

9
5041

10
15121

11
1

12
<PHONE>

13
1

14
<PHONE>

15
<PHONE>

16
<PHONE>

17
1

18
45951781075201

19
1

20
3379365788198401

21
1689515283456001

22
14079294028801

23
1

24
4454857103544668620801

25
538583682060103680001

You'll notice that if \(n\) is prime, there will be a single factorization, which hopefully makes sense. This is the factorization that only has one factor.
A very surprising fact to me is that this sequence did not show up on OEIS, which is this database that combinatorialists use to check whether or not their sequence has been studied before, and to see connections to other sequences.
To me, this just feels like the multiplicative version of the Bell numbers. The Bell numbers count how many partitions there are of a set of size \(n\). It's sequence number 110 on OEIS out of over 300,000; and this sequence just doesn't show up at all, even when I tweak it and delete the degenerate cases and so on.
I am very confused by this fact. To me, factorizations seem like an extremely natural concept, and it seems to me like it hasn't really been studied before.

This is the end of my short combinatorics talk.

Ramana Kumar: If you're willing to do it, I'd appreciate just stepping through one of the examples of the factorizations and the definition, because this is pretty new to me.
Scott: Yeah. Let's go through the first nontrivial factorization of \(\{0,1,2,3\}\):
\(\begin{split} \begin{Bmatrix} \ \ \ \{\{0,1\}, \{2,3\}\}, \ \\ \{\{0,2\}, \{1,3\}\} \end{Bmatrix} \end{split} \begin{split} \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \begin{array} { |c|c|c|c| } \hline 0 & 1 \\ \hline 2 & 3 \\ \hline \end{array} \end{split}\)
In the definition, I said a factorization should be a set of partitions such that for each way of choosing one part from each of the partitions, there will be a unique element in the intersection of those parts.
Here, I have a partition that's separating the small numbers from the large numbers: \(\{\{0,1\}, \{2,3\}\}\). And I also have a partition that's separating the even numbers from the odd numbers: \(\{\{0,2\}, \{1,3\}\}\).
And the point is that for each way of choosing either "small" or "large" and also choosing "even" or "odd", there will be a unique element of \(S\) that is the conjunction of these two choices.
In the other two nontrivial factorizations, I replace either "small and large" or "even and odd" with an "inner and outer" distinction.
David Spivak: For partitions and for many things, if I know the partitions of a set \(A\) and the partitions of a set \(B\) , then I know some partitions of \(A+B\) (the disjoint union) or I know some partitions of \(A \times B\) . Do you know any facts like that for factorizations?
Scott: Yeah. If I have two factored sets, I can get a factored set over their product, which sort of disjoint-unions the two collections of factors. For the additive thing, you're not going to get anything like that because prime sets don't have any nontrivial factorizations.

All right. I think I'm going to move on to the main talk.

(Part 2, Title Slides)   ·   ·   ·   The Main Talk (It's About Time)

(Part 2, Motivation)   ·   ·   ·   The Pearlian Paradigm

We can't talk about time without talking about Pearlian causal inference. I want to start by saying that I think the Pearlian paradigm is great. This buys me some crackpot points, but I'll say it's the best thing to happen to our understanding of time since Einstein.
I'm not going to go into all the details of Pearl's paradigm here. My talk will not be technically dependent on it; it's here for motivation.
Given a collection of variables and a joint probability distribution over those variables, Pearl can infer causal/temporal relationships between the variables. (In this talk I'm going to use "causal" and "temporal" interchangeably, though there may be more interesting things to say here philosophically.)
Pearl can infer temporal data from statistical data, which is going against the adage that "correlation does not imply causation." It's like Pearl is taking the combinatorial structure of your correlation and using that to infer causation, which I think is just really great.

Ramana Kumar: I may be wrong, but I think this is false. Or I think that that's not all Pearl needs—just the joint distribution over the variables. Doesn't he also make use of intervention distributions?
Scott: In the theory that is described in chapter two of the book Causality, he's not really using other stuff. Pearl builds up this bigger theory elsewhere. But you have some strong ability, maybe assuming simplicity or whatever (but not assuming you have access to extra information), to take a collection of variables and a joint distribution over those variables, and infer causation from correlation.
Andrew Critch: Ramana, it depends a lot on the structure of the underlying causal graph. For some causal graphs, you can actually recover them uniquely with no interventions. And only assumptions with zero-measure exceptions are needed, which is really strong.
Ramana Kumar: Right, but then the information you're using is the graph.
Andrew Critch: No, you're not. Just the joint distribution.
Ramana Kumar: Oh, okay. Sorry, go ahead.
Andrew Critch: There exist causal graphs with the property that if nature is generated by that graph and you don't know it, and then you look at the joint distribution, you will infer with probability 1 that nature was generated by that graph, without having done any interventions.
Ramana Kumar: Got it. That makes sense. Thanks.
Scott: Cool.

I am going to (a little bit) go against this, though. I'm going to claim that Pearl is kind of cheating when making this inference. The thing I want to point out is that in the sentence "Given a collection of variables and a joint probability distribution over those variables, Pearl can infer causal/temporal relationships between the variables.", the words "Given a collection of variables" are actually hiding a lot of the work.
The emphasis is usually put on the joint probability distribution, but Pearl is not inferring temporal data from statistical data alone. He is inferring temporal data from statistical data and factorization data: how the world is broken up into these variables.
I claim that this issue is also entangled with a failure to adequately handle abstraction and determinism. To point at that a little bit, one could do something like say:
"Well, what if I take the variables that I'm given in a Pearlian problem and I just forget that structure? I can just take the product of all of these variables that I'm given, and consider the space of all partitions on that product of variables that I'm given; and each one of those partitions will be its own variable. And then I can try to do Pearlian causal inference on this big set of all the variables that I get by forgetting the structure of variables that were given to me."
And the problem is that when you do that, you have a bunch of things that are deterministic functions of each other, and you can't actually infer stuff using the Pearlian paradigm.
So in my view, this cheating is very entangled with the fact that Pearl's paradigm isn't great for handling abstraction and determinism.

(Part 2, Table of Contents)   ·   ·   ·   We Can Do Better

The main thing we'll do in this talk is we're going to introduce an alternative to Pearl that does not rely on factorization data, and that therefore works better with abstraction and determinism.
Where Pearl was given a collection of variables, we are going to just consider all partitions of a given set. Where Pearl infers a directed acyclic graph, we're going to infer a finite factored set.
In the Pearlian world, we can look at the graph and read off properties of time and orthogonality/independence. A directed path between nodes corresponds to one node being before the other, and two nodes are independent if they have no common ancestor. Similarly, in our world, we will be able to read time and orthogonality off of a finite factored set.
(Orthogonality and independence are pretty similar. I'll use the word "orthogonality" when I'm talking about a combinatorial notion, and I'll use "independence" when I'm talking about a probabilistic notion.)
In the Pearlian world, d-separation, which you can read off of the graph, corresponds to conditional independence in all probability distributions that you can put on the graph. We're going to have a fundamental theorem that will say basically the same thing: conditional orthogonality corresponds to conditional independence in all probability distributions that we can put on our factored set.
In the Pearlian world, d-separation will satisfy the compositional graphoid axioms. In our world, we're just going to satisfy the compositional semigraphoid axioms. The fifth graphoid axiom is one that I claim you shouldn't have even wanted in the first place.
Pearl does causal inference. We're going to talk about how to do temporal inference using this new paradigm, and infer some very basic temporal facts that Pearl's approach can't. (Note that Pearl can also sometimes infer temporal relations that we can't—but only, from our point of view, because Pearl is making additional factorization assumptions.)
And then we'll talk about a bunch of applications.

Pearl
This Talk

A Given Collection of Variables
All Partitions of a Given Set

Directed Acyclic Graph
Finite Factored Set

Directed Path Between Nodes
"Time"

No Common Ancestor
"Orthogonality"

d-Separation
"Conditional Orthogonality"

Compositional Graphoid
Compositional Semigraphoid

d-Separation  Conditional Independence
The Fundamental Theorem

Causal Inference
Temporal Inference

Many Many Applications
Many Many Applications

Excluding the motivation, table of contents, and example sections, this table also serves as an outline of the two talks. We've already talked about set partitions and finite factored sets, so now we're going to talk about time and orthogonality.

(Part 2, Main Body)   ·   ·   ·   Time and Orthogonality

I think that if you capture one definition from this second part of the talk, it should be this one. Given a finite factored set as context, we're going to define the history of a partition.
Let \(F = (S,B)\) be a finite factored set. And let \(X, Y \in \mathrm{Part}(S)\) be partitions of \(S\).
The history of \(X\) , written \(h^F(X)\), is the smallest set of factors \(H \subseteq B\) such that for all \(s, t \in S\), if \(s \sim_b t\) for all \(b \in H\), then \(s \sim_X t\).
The history of \(X\) , then, is the smallest set of factors \(H\) —so, the smallest subset of \(B\) —such that if I take an element of \(S\) and I hide it from you, and you want to know which part in \(X\) it is in, it suffices for me to tell you which part it is in within each of the factors in \(H\) .
So the history \(H\) is a set of factors of \(S\) , and knowing the values of all the factors in \(H\) is sufficient to know the value of \(X\) , or to know which part in \(X\) a given element is going to be in. I'll give an example soon that will maybe make this a little more clear.
We're then going to define time from history. We'll say that \(X\) is weakly before \(Y\), written \(X\leq^F Y\), if \(h^F(X)\subseteq h^F(Y)\) . And we'll say that \(X\) is strictly before \(Y\), written \(X<^F Y\), if \(h^F(X)\subset h^F(Y)\).
One analogy one could draw is that these histories are like the past light cones of a point in spacetime. When one point is before another point, then the backwards light cone of the earlier point is going to be a subset of the backwards light cone of the later point. This helps show why "before" can be like a subset relation.
We're also going to define orthogonality from history. We'll say that two partitions \(X\) and \(Y\) are orthogonal, written \(X\perp^FY\) , if their histories are disjoint: \(h^F(X)\cap h^F(Y)=\{\}\).
Now I'm going to go through an example.

(Part 2, Examples)   ·   ·   ·   Game of Life

Let \(S\) be the set of all Game of Life computations starting from an \([-n,n]\times[-n,n]\) board.
Let \(R=\{(r,c,t)\in\mathbb{Z}^3\mid0\leq t\leq n,\ \) \(|r|\leq n-t,\ |c|\leq n-t\}\) (i.e., cells computable from the initial \([-n,n]\times[-n,n]\) board). For \((r,c,t)\in R\), let \(\ell(r,c,t)\subseteq S\) be the set of all computations such that the cell at row \(r\) and column \(c\) is alive at time \(t\).
(Minor footnote: I've done some small tricks here in order to deal with the fact that the Game of Life is normally played on an infinite board. We want to deal with the finite case, and we don't want to worry about boundary conditions, so we're only going to look at the cells that are uniquely determined by the initial board. This means that the board will shrink over time, but this won't matter for our example.)
\(S\) is the set of all Game of Life computations, but since the Game of Life is deterministic, the set of all computations is in bijective correspondence with the set of all initial conditions. So \(|S|=2^{(2n+1)^{2}}\) , the number of initial board states.
This also gives us a nice factorization on the set of all Game of Life computations. For each cell, there's a partition that separates out the Game of Life computations in which that cell is alive at time 0 from the ones where it's dead at time 0. Our factorization, then, will be a set of \((2n+1)^{2}\) binary factors, one for each question of "Was this cell alive or dead at time 0?".
Formally: For \((r,c,t)\in R\), let \(L_{(r,c,t)}=\{\ell(r,c,t),S\setminus \ell(r,c,t)\}\). Let \(F=(S,B)\), where \(B=\{L_{(r,c,0)}\mid -n\leq r,c\leq n\}\).
There will also be other partitions on this set of all Game of Life computations that we can talk about. For example, you can take a cell and a time \(t\) and say, "Is this cell alive at time \(t\)?", and there will be a partition that separates out the computations where that cell is alive at time \(t\) from the computations where it's dead at time \(t\).
Here's an example of that:

The lowest grid shows a section of the initial board state.
The blue, green, and red squares on the upper boards are (cell, time) pairs. Each square corresponds to a partition of the set of all Game of Life computations, "Is that cell alive or dead at the given time \(t\)?"
The history of that partition is going to be all the cells in the initial board that go into computing whether the cell is alive or dead at time \(t\) . It's everything involved in figuring out that cell's state. E.g., knowing the state of the nine light-red cells in the initial board always tells you the state of the red cell in the second board.
In this example, the partition corresponding to the red cell's state is strictly before the partition corresponding to the blue cell. The question of whether the red cell is alive or dead is before the question of whether the blue cell is alive or dead.
Meanwhile, the question of whether the red cell is alive or dead is going to be orthogonal to the question of whether the green cell is alive or dead.
And the question of whether the blue cell is alive or dead is not going to be orthogonal to the question of whether the green cell is alive or dead, because they intersect on the cyan cells.
Generalizing the point, fix \(X=L_{(r_X,c_X,t_X)}, Y=L_{(r_Y,c_Y,t_Y)}\), where \((r_X,c_X,t_X),(r_Y,c_Y,t_Y)\in R\). Then:

\(h^{F}(X)=\{L_{(r,c,0)}\in B\mid |r_X-r|\leq t_X,|c_X-c|\leq t_X\}\).
\(X \ ᐸ^{F} \ Y\) if and only if \(t_X \ ᐸ \ t_Y\) and \(|r_Y-r_X|,|c_Y-c_X|\leq t_Y-t_X\).
\(X \perp^F Y\) if and only if \(|r_Y-r_X|> t_Y+t_X\) or \(|c_Y-c_X|> t_Y+t_X\).

We can also see that the blue and green cells look almost orthogonal. If we condition on the values of the two cyan cells in the intersection of their histories, then the blue and green partitions become orthogonal. That's what we're going to discuss next.

David Spivak: A priori, that would be a gigantic computation—to be able to tell me that you understand the factorization structure of that Game of Life. So what intuition are you using to be able to make that claim, that it has the kind of factorization structure you're implying there?
Scott: So, I've defined the factorization structure.
David Spivak: You gave us a certain factorization already. So somehow you have a very good intuition about history, I guess. Maybe that's what I'm asking about.
Scott: Yeah. So, if I didn't give you the factorization, there's this obnoxious number of factorizations that you could put on the set here. And then for the history, the intuition I'm using is: "What do I need to know in order to compute this value?"
I actually went through and I made little gadgets in Game of Life to make sure I was right here, that every single cell actually could in some situations affect the cells in question. But yeah, the intuition that I'm working from is mostly about the information in the computation. It's "Can I construct a situation where if only I knew this fact, I would be able to compute what this value is? And if I can't, then it can take two different values."
David Spivak: Okay. I think deriving that intuition from the definition is something I'm missing, but I don't know if we have time to go through that.
Scott: Yeah, I think I'm not going to here.

(Part 2, Main Body)   ·   ·   ·   Conditional Orthogonality

So, just to set your expectations: Every time I explain Pearlian causal inference to someone, they say that d-separation is the thing they can't remember. d-separation is a much more complicated concept than "directed paths between nodes" and "nodes without any common ancestors" in Pearl; and similarly, conditional orthogonality will be much more complicated than time and orthogonality in our paradigm. Though I do think that conditional orthogonality has a much simpler and nicer definition than d-separation.
We'll begin with the definition of conditional history. We again have a fixed finite set as our context. Let \(F=(S,B)\) be a finite factored set, let \(X,Y,Z\in\text{Part}(S)\), and let \(E\subseteq S\).
The conditional history of \(X\) given \(E\), written \(h^F(X|E)\), is the smallest set of factors \(H\subseteq B\) satisfying the following two conditions:

For all \(s,t\in E\), if \(s\sim_{b} t\) for all \(b\in H\), then \(s\sim_X t\).
For all \(s,t\in E\) and \(r\in S\), if \(r\sim_{b_0} s\) for all \(b_0\in H\) and \(r\sim_{b_1} t\) for all \(b_1\in B\setminus H\), then \(r\in E\).

The first condition is much like the condition we had in our definition of history, except we're going to make the assumption that we're in \(E\). So the first condition is: if all you know about an object is that it's in \(E\), and you want to know which part it's in within \(X\), it suffices for me to tell you which part it's in within each factor in the history \(H\).
Our second condition is not actually going to mention \(X\). It's going to be a relationship between \(E\) and \(H\). And it says that if you want to figure out whether an element of \(S\) is in \(E\), it's sufficient to parallelize and ask two questions:

"If I only look at the values of the factors in \(H\), is 'this point is in \(E\)' compatible with that information?"
"If I only look at the values of the factors in \(B\setminus H\) , is 'this point is in \(E\)' compatible with that information?"

If both of these questions return "yes", then the point has to be in \(E\).
I am not going to give an intuition about why this needs to be a part of the definition. I will say that without this second condition, conditional history would not even be well-defined, because it wouldn't be closed under intersection. And so I wouldn't be able to take the smallest set of factors in the subset ordering.
Instead of justifying this definition by explaining the intuitions behind it, I'm going to justify it by using it and appealing to its consequences.
We're going to use conditional history to define conditional orthogonality, just like we used history to define orthogonality. We say that \(X\) and \(Y\) are orthogonal given \(E\subseteq S\), written \(X \perp^{F} Y \mid E\), if the history of \(X\) given \(E\) is disjoint from the history of \(Y\) given \(E\): \(h^F(X|E)\cap h^F(Y|E)=\{\}\).
We say \(X\) and \(Y\) are orthogonal given \(Z\in\text{Part}(S)\), written \(X \perp^{F} Y \mid Z\), if \(X \perp^{F} Y \mid z\) for all \(z\in Z\). So what it means to be orthogonal given a partition is just to be orthogonal given each individual way that the partition might be, each individual part in that partition.
I've been working with this for a while and it feels pretty natural to me, but I don't have a good way to push the naturalness of this condition. So again, I instead want to appeal to the consequences.

(Part 2, Main Body)   ·   ·   ·   Compositional Semigraphoid Axioms

Conditional orthogonality satisfies the compositional semigraphoid axioms, which means finite factored sets are pretty well-behaved.
Let \(F=(S,B)\) be a finite factored set, and let \(X,Y,Z,W\in \text{Part}(S)\) be partitions of \(S\). Then:

If \(X \perp^{F} Y \mid Z\), then \(Y \perp^{F} X \mid Z\).   (symmetry)
If \(X \perp^{F} (Y\vee_S W) \mid Z\), then \(X \perp^{F} Y \mid Z\) and \(X \perp^{F} W \mid Z\).   (decomposition)
If \(X \perp^{F} (Y\vee_S W) \mid Z\), then \(X \perp^{F} Y \mid (Z\vee_S W)\).   (weak union)
If \(X \perp^{F} Y \mid Z\) and \(X \perp^{F} W \mid (Z\vee_S Y)\), then \(X \perp^{F} (Y\vee_S W) \mid Z\).   (contraction)
If \(X \perp^{F} Y \mid Z\) and If \(X \perp^{F} W \mid Z\), then \(X \perp^{F} (Y\vee_S W) \mid Z\).   (composition)

The first four properties here make up the semigraphoid axioms, slightly modified because I'm working with partitions rather than sets of variables, so union is replaced with common refinement. There's another graphoid axiom which we're not going to satisfy; but I argue that we don't want to satisfy it, because it doesn't play well with determinism.
The fifth property here, composition, is maybe one of the most unintuitive, because it's not exactly satisfied by probabilistic independence.
Decomposition and composition act like converses of each other. Together, conditioning on \(Z\) throughout, they say that \(X\) is orthogonal to both \(Y\) and \(W\) if and only if \(X\) is orthogonal to the common refinement of \(Y\) and \(W\).

(Part 2, Main Body)   ·   ·   ·   The Fundamental Theorem

In addition to being well-behaved, I also want to show that conditional orthogonality is pretty powerful. The way I want to do this is by showing that conditional orthogonality exactly corresponds to conditional independence in all probability distributions you can put on your finite factored set. Thus, much like d-separation in the Pearlian picture, conditional orthogonality can be thought of as a combinatorial version of probabilistic independence.
A probability distribution on a finite factored set \(F=(S,B)\) is a probability distribution \(P\) on \(S\) that can be thought of as coming from a bunch of independent probability distributions on each of the factors in \(B\). So \(P(s)=\prod_{b\in B}P([s]_b)\) for all \(s\in S\).
This effectively means that your probability distribution factors the same way your set factors: the probability of any given element is the product of the probabilities of each of the individual parts that it's in within each factor.
The fundamental theorem of finite factored sets says: Let \(F=(S,B)\) be a finite factored set, and let \(X,Y,Z\in \text{Part}(S)\) be partitions of \(S\). Then \(X \perp^{F} Y \mid Z\) if and only if for all probability distributions \(P\) on \(F\), and all \(x\in X\), \(y\in Y\), and \(z\in Z\), we have \(P(x\cap z)\cdot P(y\cap z)= P(x\cap y\cap z)\cdot P(z)\). I.e., \(X\) is orthogonal to \(Y\) given \(Z\) if and only conditional independence is satisfied across all probability distributions.
This theorem, for me, was a little nontrivial to prove. I had to go through defining certain polynomials associated with the subsets, and then dealing with unique factorization in the space of these polynomials; I think the proof was eight pages or something.
The fundamental theorem allows us to infer orthogonality data from probabilistic data. If I have some empirical distribution, or I have some Bayesian distribution, I can use that to infer some orthogonality data. (We could also imagine orthogonality data coming from other sources.) And then we can use this orthogonality data to get temporal data.
So next, we're going to talk about how to get temporal data from orthogonality data.

(Part 2, Main Body)   ·   ·   ·   Temporal Inference

We're going to start with a finite set \(\Omega\), which is our sample space.
One naive thing that you might think we would try to do is infer a factorization of \(\Omega\). We're not going to do that because that's going to be too restrictive. We want to allow for \(\Omega\) to maybe hide some information from us, for there to be some latent structure and such.
There may be some situations that are distinct without being distinct in \(\Omega\). So instead, we're going to infer a factored set model of \(\Omega\): some other set \(S\), and a factorization of \(S\), and a function from \(S\) to \(\Omega\).
A model of \(\Omega\) is a pair \((F, f)\), where \(F=(S,B)\) is a finite factored set and \(f:S\rightarrow \Omega\). ( \(f\) need not be injective or surjective.)
Then if I have a partition of \(\Omega\), I can send this partition backwards across \(f\) and get a unique partition of \(S\). If \(X\in \text{Parts}(\Omega)\), then \(f^{-1}(X)\in \text{Parts}(S)\) is given by \(s\sim_{f^{-1}(X)}t\Leftrightarrow f(s)\sim_X f(t)\).
Then what we're going to do is take a bunch of orthogonality facts about \(\Omega\), and we're going to try to find a model which captures the orthogonality facts.
We will take as given an orthogonality database on \(\Omega\), which is a pair \(D = (O, N)\), where \(O\) (for "orthogonal") and \(N\) (for "not orthogonal") are each sets of triples \((X,Y,Z)\) of partitions of \(\Omega\). We'll think of these as rules about orthogonality.
What it means for a model \((F,f)\) to satisfy a database \(D\) is:

\(f^{-1}(X) \perp^{F} f^{-1}(Y) \mid f^{-1}(Z)\) whenever \((X,Y,Z)\in O\), and
\(\) \(\lnot (f^{-1}(X) \perp^{F} f^{-1}(Y) \mid f^{-1}(Z))\) whenever \((X,Y,Z)\in N\).

So we have these orthogonality rules we want to satisfy, and we want to consider the space of all models that are consistent with these rules. And even though there will always be infinitely many models that are consistent with my database, if at least one is—you can always just add more information that you then delete with \(f\)—we would like to be able to sometimes infer that for all models that satisfy our database, \(f^{-1}(X)\) is before \(f^{-1}(Y)\).
And this is what we're going to mean by inferring time. If all of our models \((F,f)\) that are consistent with the database \(D\) satisfy some claim about time \(f^{-1}(X) \ ᐸ^F \ f^{-1}(Y)\), we'll say that \(X \ ᐸ_D \ Y\).

(Part 2, Examples)   ·   ·   ·   Two Binary Variables (Pearl)

So we've set up this nice combinatorial notion of temporal inference. The obvious next questions are:

Can we actually infer interesting facts using this method, or is it vacuous?
And: How does this framework compare to Pearlian temporal inference?

Pearlian temporal inference is really quite powerful; given enough data, it can infer temporal sequence in a wide variety of situations. How powerful is the finite factored sets approach by comparison?
To address that question, we'll go to an example. Let \(X\) and \(Y\) be two binary variables. Pearl asks: "Are \(X\) and \(Y\) independent?" If yes, then there's no path between the two. If no, then there may be a path from \(X\) to \(Y\), or from \(Y\) to \(X\), or from a third variable to both \(X\) and \(Y\).
In either case, we're not going to infer any temporal relationships.
To me, it feels like this is where the adage "correlation does not imply causation" comes from. Pearl really needs more variables in order to be able to infer temporal relationships from more rich combinatorial structures.
However, I claim that this Pearlian ontology in which you're handed this collection of variables has blinded us to the obvious next question, which is: is \(X\) independent of \(X \ \mathrm{XOR} \ Y\)?
In the Pearlian world, \(X\) and \(Y\) were our variables, and \(X \ \mathrm{XOR} \ Y\) is just some random operation on those variables. In our world, \(X \ \mathrm{XOR} \ Y\) instead is a variable on the same footing as \(X\) and \(Y\). The first thing I do with my variables \(X\) and \(Y\) is that I take the product \(X \times Y\) and then I forget the labels \(X\) and \(Y\).
So there's this question, "Is \(X\) independent of \(X \ \mathrm{XOR} \ Y\)?". And if \(X\) is independent of \(X \ \mathrm{XOR} \ Y\), we're actually going to be able to conclude that \(X\) is before \(Y\)!
So not only is the finite factored set paradigm non-vacuous, and not only is it going to be able to keep up with Pearl and infer things Pearl can't, but it's going to be able to infer a temporal relationship from only two variables.
So let's go through the proof of that.

(Part 2, Examples)   ·   ·   ·   Two Binary Variables (Factored Sets)

Let \(\Omega=\{00,01,10,11\}\), and let \(X\), \(Y\), and \(Z\) be the partitions (/questions):

\(X = \{\{00,01\}, \{10,11\}\}\).   (What is the first bit?)
\(Y=\{\{00,10\}, \{01,11\}\}\).   (What is the second bit?)
\(Z=\{\{00,11\}, \{01,10\}\}\).   (Do the bits match?)

Let \(D = (O,N)\), where \(O = \{(X, Z, \{\Omega\})\}\) and \(N = \{(Z, Z, \{\Omega\})\}\). If we'd gotten this orthogonality database from a probability distribution, then we would have more than just two rules, since we would observe more orthogonality and non-orthogonality than that. But temporal inference is monotonic with respect to adding more rules, so we can just work with the smallest set of rules we'll need for the proof.
The first rule says that \(X\) is orthogonal to \(Z\). The second rule says that \(Z\) is not orthogonal to itself, which is basically just saying that \(Z\) is non-deterministic; it's saying that both of the parts in \(Z\) are possible, that both are supported under the function \(f\). The \(\{\Omega\}\) indicates that we aren't making any conditions.
From this, we'll be able to prove that \(X \ ᐸ_D \ Y\).

Proof. First, we'll show that that \(X\) is weakly before \(Y\). Let \((F,f)\) satisfy \(D\). Let \(H_X\) be shorthand for \(h^F(f^{-1}(X))\), and likewise let \(H_Y=h^F(f^{-1}(Y))\) and \(H_Z=h^F(f^{-1}(Z))\).
Since \((X,Z,\{\Omega\})\in O\), we have that \(H_X\cap H_Z=\{\}\); and since \((Z,Z,\{\Omega\})\in N\), we have that \(H_Z\neq \{\}\).
Since \(X\leq_{\Omega} Y\vee_{\Omega} Z\)—that is, since \(X\) can be computed from \(Y\) together with \(Z\)—\(H_X\subseteq H_Y\cup H_Z\). (Because a partition's history is the smallest set of factors needed to compute that partition.)
And since \(H_X\cap H_Z=\{\}\), this implies \(H_X\subseteq H_Y\), so \(X\) is weakly before \(Y\).
To show the strict inequality, we'll assume for the purpose of contradiction that \(H_X\) = \(H_Y\).
Notice that \(Z\) can be computed from \(X\) together with \(Y\)—that is, \(Z\leq_{\Omega} X\vee_{\Omega} Y\)—and therefore \(H_Z\subseteq H_X\cup H_Y\) (i.e., \(H_Z \subseteq H_X\) ). It follows that \(H_Z = (H_X\cup H_Y)\cap H_Z=H_X\cap H_Z\). But since \(H_Z\) is also disjoint from \(H_X\), this means that \(H_Z = \{\}\), a contradiction.
Thus \(H_X\neq H_Y\), so \(H_X \subset H_Y\), so \(f^{-1}(X) \ ᐸ^F \ f^{-1}(Y)\), so \(X \ ᐸ_D \ Y\). □

When I'm doing temporal inference using finite factored sets, I largely have proofs that look like this. We collect some facts about emptiness or non-emptiness of various Boolean combinations of histories of variables, and we use these to conclude more facts about histories of variables being subsets of each other.
I have a more complicated example that uses conditional orthogonality, not just orthogonality; I'm not going to go over it here.
One interesting point I want to make here is that we're doing temporal inference—we're inferring that \(X\) is before \(Y\)—but I claim that we're also doing conceptual inference.
Imagine that I had a bit, and it's either a 0 or a 1, and it's either blue or green. And these two facts are primitive and independently generated. And I also have this other concept that's like, "Is it grue or bleen?", which is the \(\mathrm{XOR}\) of blue/green and 0/1.
There's a sense in which we're inferring \(X\) is before \(Y\) , and in that case, we can infer that blueness is before grueness. And that's pointing at the fact that blueness is more primitive, and grueness is a derived property.
In our proof, \(X\) and \(Z\) can be thought of as these primitive properties, and \(Y\) is a derived property that we're getting from them. So we're not just inferring time; we're inferring facts about what are good, natural concepts. And I think that there's some hope that this ontology can do for the statement "you can't really distinguish between blue and grue" what Pearl can do to the statement "correlation does not imply causation".

(Part 2, Main Body)   ·   ·   ·   Applications / Future Work / Speculation

The future work I'm most excited by with finite factored sets falls into three rough categories: inference (which involves more computational questions), infinity (more mathematical), and embedded agency (more philosophical).
Research topics related to inference:

Decidability of Temporal Inference
Efficient Temporal Inference
Conceptual Inference
Temporal Inference from Raw Data and Fewer Ontological Assumptions
Temporal Inference with Deterministic Relationships
Time without Orthogonality
Conditioned Factored Sets

There are a lot of research directions suggested by questions like "How do we do efficient inference in this paradigm?". Some of the questions here come from the fact that we're making fewer assumptions than Pearl, and are in some sense more coming from the raw data.
Then I have the applications that are about extending factored sets to the infinite case:

Extending Definitions to the Infinite Case
The Fundamental Theorem of Finite-Dimensional Factored Sets
Continuous Time
New Lens on Physics

Everything I've presented in this talk was under the assumption of finiteness. In some cases this wasn't necessary—but in a lot of cases it actually was, and I didn't draw attention to this.
I suspect that the fundamental theorem can be extended to finite-dimensional factored sets (i.e., factored sets where \(|B|\) is finite), but it can not be extended to arbitrary-dimension factored sets.
And then, what I'm really excited about is applications to embedded agency:

Embedded Observations
Counterfactability
Cartesian Frames Successor
Unraveling Causal Loops
Conditional Time
Logical Causality from Logical Induction
Orthogonality as Simplifying Assumptions for Decisions
Conditional Orthogonality as Abstraction Desideratum

I focused on the temporal inference aspect of finite factored sets in this talk, because it's concrete and tangible to be able to say, "Ah, we can do Pearlian temporal inference, only we can sometimes infer more structure and we rely on fewer assumptions."
But really, a lot of the applications I'm excited about involve using factored sets to model situations, rather than inferring factored sets from data.
Anywhere that we currently model a situation using graphs with directed edges that represent information flow or causality, we might instead be able to use factored sets to model the situation; and this might allow our models to play more nicely with abstraction.
I want to build up the factored set ontology as an alternative to graphs when modeling agents interacting with things, or when modeling information flow. And I'm really excited about that direction.

The post Finite Factored Sets appeared first on Machine Intelligence Research Institute.

Finite Factored Sets


May 2021 Newsletter

MIRI senior researcher Scott Garrabrant has a major new result, "Finite Factored Sets," that he'll be unveiling in an online talk this Sunday at noon Pacific time. (Zoom link.) For context on the result, see Scott's new post "Saving Time."
In other big news, MIRI has just received its two largest individual donations of all time! Ethereum inventor Vitalik Buterin has donated ~$4.3 million worth of ETH to our research program, while an anonymous long-time supporter has donated MKR tokens we liquidated for an astounding ~$15.6 million. The latter donation is restricted so that we can spend a maximum of $2.5 million of it per year until 2025, like a multi-year grant.
Both donors have our massive thanks for these incredible gifts to support our work!
Other MIRI updates

Mark Xu and Evan Hubinger use "Cartesian world models" to distinguish "consequential agents" (which assign utility to environment states, internal states, observations, and/or actions) "structural agents" (which optimize "over the set of possible decide functions instead of the set of possible actions"), and "conditional agents" (which map e.g. environmental states to utility functions, rather than mapping them to utility).
In Gradations of Inner Alignment Obstacles, Abram Demski makes three "contentious claims":

The most useful definition of "mesa-optimizer" doesn't require them to perform explicit search, contrary to the current standard.
Success at aligning narrowly superhuman models might be bad news.
Some versions of the lottery ticket hypothesis seem to imply that randomly initialized networks already contain deceptive agents.

Eliezer Yudkowsky comments on the relationship between early AGI systems' alignability and capabilities.

News and links

John Wentworth announces a project to test the natural abstraction hypothesis, which asserts that "most high-level abstract concepts used by humans are 'natural'" and therefore "a wide range of architectures will reliably learn similar high-level concepts".
Open Philanthropy's Joe Carlsmith asks "Is Power-Seeking AI an Existential Risk?", and Luke Muehlhauser asks for examples of treacherous turns in the wild (also on LessWrong).
From DeepMind's safety researchers: What Mechanisms Drive Agent Behavior?, Alignment of Language Agents, and An EPIC Way to Evaluate Reward Functions. Also, Rohin Shah provides his advice on entering the field.
Owen Shen and Peter Hase summarize 70 recent papers on model transparency, interpretability, and explainability.
Eli Tyre asks: How do we prepare for final crunch time? (I would add some caveats: Some roles and scenarios imply that you'll have less impact on the eve of AGI, and can have far more impact today. For some people, "final crunch time" may be now, and marginal efforts matter less later. Further, some forms of "preparing for crunch time" will fail if there aren't clear warning shots or fire alarms.)
Paul Christiano launches a new organization that will be his focus going forward: the Alignment Research Center. Learn more about Christiano's research approach in My Research Methodology and in his recent AMA.

The post May 2021 Newsletter appeared first on Machine Intelligence Research Institute.

May 2021 Newsletter


Saving Time

Note: This is a preamble to Finite Factored Sets, a sequence I'll be posting over the next few weeks. This Sunday at noon Pacific time, I'll be giving a Zoom talk (link) introducing Finite Factored Sets, a framework which I find roughly as technically interesting as logical induction.
(Update May 25: A video and blog post introducing Finite Factored Sets is now available here.)

For the last few years, a large part of my research motivation has been directed at trying to save the concept of time—save it, for example, from all the weird causal loops created by decision theory problems. This post will hopefully explain why I care so much about time, and what I think needs to be fixed.

Why Time?
My best attempt at a short description of time is that time is causality. For example, in a Pearlian Bayes net, you draw edges from earlier nodes to later nodes. To the extent that we want to think about causality, then, we will need to understand time.
Importantly, time is the substrate in which learning and commitments take place. When agents learn, they learn over time. The passage of time is like a ritual in which opportunities are destroyed and knowledge is created. And I think that many models of learning are subtly confused, because they are based on confused notions of time.
Time is also crucial for thinking about agency. My best short-phrase definition of agency is that agency is time travel. An agent is a mechanism through which the future is able to affect the past. An agent models the future consequences of its actions, and chooses actions on the basis of those consequences. In that sense, the consequence causes the action, in spite of the fact that the action comes earlier in the standard physical sense.

Problem: Time is Loopy
The main thing going wrong with time is that it is "loopy."
The primary confusing thing about Newcomb's problem is that we want to think of our decision as coming "before" the filling of the boxes, in spite of the fact that it physically comes after. This is hinting that maybe we want to understand some other "logical" time in addition to the time of physics.
However, when we attempt to do this, we run into two problems: Firstly, we don't understand where this logical time might come from, or how to learn it, and secondly, we run into some apparent temporal loops.
I am going to set aside the first problem and focus on the second.
The easiest way to see why we run into temporal loops is to notice that it seems like physical time is at least a little bit entangled with logical time.
Imagine the point of view of someone running a physics simulation of Newcomb's problem, and tracking all of the details of all of the atoms. From that point of view, it seems like there is a useful sense in which the filling of the boxes comes before an agent's decision to one-box or two-box. At the same time, however, those atoms compose an agent that shouldn't make decisions as though it were helpless to change anything.
Maybe the solution here is to think of there being many different types of "before" and "after," "cause" and "effect," etc. For example, we could say that X is before Y from an agent-first perspective, but Y is before X from a physics-first perspective.
I think this is right, and we want to think of there as being many different systems of time (hopefully predictably interconnected). But I don't think this resolves the whole problem.
Consider a pair of FairBot agents that successfully execute a Löbian handshake to cooperate in an open-source prisoner's dilemma. I want to say that each agent's cooperation causes the other agent's cooperation in some sense. I could say that relative to each agent the causal/temporal ordering goes a different way, but I think the loop is an important part of the structure in this case. (I also am not even sure which direction of time I would want to associate with which agent.)
We also are tempted to put loops in our time/causality for other reasons. For example, when modeling a feedback loop in a system that persists over time, we might draw structures that look a lot like a Bayes net, but are not acyclic (e.g., a POMDP). We could think of this as a projection of another system that has an extra dimension of time, but it is a useful projection nonetheless.

Solution: Abstraction
My main hope for recovering a coherent notion of time and unraveling these temporal loops is via abstraction.
In the example where the agent chooses actions based on their consequences, I think that there is an abstract model of the consequences that comes causally before the choice of action, which comes before the actual physical consequences.
In Newcomb's problem, I want to say that there is an abstract model of the action that comes causally before the filling of the boxes.
In the open source prisoners' dilemma, I want to say that there is an abstract proof of cooperation that comes causally before the actual program traces of the agents.
All of this is pointing in the same direction: We need to have coarse abstract versions of structures come at a different time than more refined versions of the same structure. Maybe when we correctly allow for different levels of description having different links in the causal chain, we can unravel all of the time loops.

But How?
Unfortunately, our best understanding of time is Pearlian causality, and Pearlian causality does not do great with abstraction.
Pearl has Bayes nets with a bunch of variables, but when some of those variables are coarse abstract versions of other variables, then we have to allow for determinism, since some of our variables will be deterministic functions of each other; and the best parts of Pearl do not do well with determinism.
But the problem runs deeper than that. If we draw an arrow in the direction of the deterministic function, we will be drawing an arrow of time from the more refined version of the structure to the coarser version of that structure, which is in the opposite direction of all of our examples.
Maybe we could avoid drawing this arrow from the more refined node to the coarser node, and instead have a path from the coarser node to the refined node. But then we could just make another copy of the coarser node that is deterministically downstream of the more refined node, adding no new degrees of freedom. What is then stopping us from swapping the two copies of the coarser node?
Overall, it seems to me that Pearl is not ready for some of the nodes to be abstract versions of other nodes, which I think needs to be fixed in order to save time.

Discussion on: LessWrong

The post Saving Time appeared first on Machine Intelligence Research Institute.

Saving Time


Our all-time largest donation, and major crypto support from Vitalik Buterin

I'm thrilled to announce two major donations to MIRI!

First, a long-time supporter has given MIRI by far our largest donation ever: $2.5 million per year over the next four years, and an additional ~$5.6 million in 2025.
This anonymous donation comes from a cryptocurrency investor who previously donated $1.01M in ETH to MIRI in 2017. Their amazingly generous new donation comes in the form of 3001 MKR, governance tokens used in MakerDAO, a stablecoin project on the Ethereum blockchain. MIRI liquidated the donated MKR for $15,592,829 after receiving it. With this donation, the anonymous donor becomes our largest all-time supporter.
This donation is subject to a time restriction whereby MIRI can spend a maximum of $2.5M of the gift in each of the next four calendar years, 2021–2024. The remaining $5,592,829 becomes available in 2025.

Second, in other amazing news, the inventor and co-founder of Ethereum, Vitalik Buterin, yesterday gave us a surprise donation of 1050 ETH, worth $4,378,159.
This is the third-largest contribution to MIRI's research program to date, after Open Philanthropy's ~$7.7M grant in 2020 and the anonymous donation above.
Vitalik has previously donated over $1M to MIRI, including major support in our 2017 fundraiser.

We're beyond grateful for these two unprecedented individual gifts! Both donors have our heartfelt thanks.

The post Our all-time largest donation, and major crypto support from Vitalik Buterin appeared first on Machine Intelligence Research Institute.


April 2021 Newsletter

MIRI updates

MIRI researcher Abram Demski writes regarding counterfactuals:

I've felt like the problem of counterfactuals is "mostly settled" (modulo some math working out) for about a year, but I don't think I've really communicated this online. Partly, I've been waiting to write up more formal results. But other research has taken up most of my time, so I'm not sure when I would get to it.
So, the following contains some "shovel-ready" problems. If you're convinced by my overall perspective, you may be interested in pursuing some of them. I think these directions have a high chance of basically solving the problem of counterfactuals (including logical counterfactuals). […]

Alex Mennen writes a thoughtful critique of one of the core arguments behind Abram's new take on counterfactuals; Abram replies.
Abram distinguishes simple Bayesians (who reason according to the laws of probability theory) from reflective Bayesians (who endorse background views that justify Bayesianism), and argues that simple Bayesians can better "escape the trap" of traditional issues with Bayesian reasoning.
Abram explains the motivations behind his learning normativity research agenda, providing "four different elevator pitches, which tell different stories" about how the research agenda's desiderata hang together.

News and links

CFAR co-founder Julia Galef has an excellent new book out on human rationality and motivated reasoning: The Scout Mindset: Why Some People See Things Clearly and Others Don't.
Katja Grace argues that there is pressure for systems with preferences to become more coherent, efficient, and goal-directed.
Andrew Critch discusses multipolar failure scenarios and "multi-agent processes with a robust tendency to play out irrespective of which agents execute which steps in the process".
A second AI alignment podcast joins Daniel Filan's AI X-Risk Research Podcast: Quinn Dougherty's Technical AI Safety Podcast, with a recent episode featuring Alex Turner.
A simple but important observation by Mark Xu: Strong Evidence is Common.

The post April 2021 Newsletter appeared first on Machine Intelligence Research Institute.

April 2021 Newsletter


March 2021 Newsletter

MIRI updates

MIRI's Eliezer Yudkowsky and Evan Hubinger comment in some detail on Ajeya Cotra's The Case for Aligning Narrowly Superhuman Models. This conversation touches on some of the more important alignment research views at MIRI, such as the view that alignment requires a thorough understanding of AGI systems' reasoning "under the hood", and the view that early AGI systems should most likely avoid human modeling if possible.
From Eliezer Yudkowsky: A Semitechnical Introductory Dialogue on Solomonoff Induction. (Also discussed by Richard Ngo.)
MIRI research associate Vanessa Kosoy discusses infra-Bayesianism on the AI X-Risk Research Podcast.
Eliezer Yudkowsky and Chris Olah discuss ML transparency on social media.

News and links

Brian Christian, author of The Alignment Problem: Machine Learning and Human Values, discusses his book on the 80,000 Hours Podcast.
Chris Olah's team releases Multimodal Neurons in Artificial Neural Networks, on artificial neurons that fire for multiple conceptually related stimuli.
Vitalik Buterin reflects on Inadequate Equilibria's arguments in the course of discussing prediction market inefficiencies.

The post March 2021 Newsletter appeared first on Machine Intelligence Research Institute.

March 2021 Newsletter


February 2021 Newsletter

MIRI updates

Abram Demski distinguishes different versions of the problem of "pointing at" human values in AI alignment.
Evan Hubinger discusses "Risks from Learned Optimization" on the AI X-Risk Research Podcast.
Eliezer Yudkowsky comments on AI safety via debate and Goodhart's law.
MIRI supporters donated ~$135k on Giving Tuesday, of which ~26% was matched by Facebook and ~28% by employers for a total of $207,436! MIRI also received $6,624 from TisBest Philanthropy in late December, largely through Round Two of Ray Dalio's #RedefineGifting initiative. Our thanks to all of you!
Spencer Greenberg discusses society and education with Anna Salamon and Duncan Sabien on the Clearer Thinking podcast.
We Want MoR: Eliezer participates in a (spoiler-laden) discussion of Harry Potter and the Methods of Rationality.

News and links

Richard Ngo reflects on his time in effective altruism:
[…] Until recently, I was relatively passive in making big decisions. Often that meant just picking the most high-prestige default option, rather than making a specific long-term plan. This also involved me thinking about EA from a "consumer" mindset rather than a "producer" mindset. When it seemed like something was missing, I used to wonder why the people responsible hadn't done it; now I also ask why I haven't done it, and consider taking responsibility myself.
Partly that's just because I've now been involved in EA for longer. But I think I also used to overestimate how established and organised EA is. In fact, we're an incredibly young movement, and we're still making up a lot of things as we go along. That makes proactivity more important.
Another reason to value proactivity highly is that taking the most standard route to success is often overrated. […] My inspiration in this regard is a friend of mine who has, three times in a row, reached out to an organisation she wanted to work for and convinced them to create a new position for her.

Ngo distinguishes claims about goal specification, orthogonality, instrumental convergence, value fragility, and Goodhart's law based on whether they refer to systems at training time versus deployment time.
Connor Leahy, author of The Hacker Learns to Trust, argues (among other things) that "GPT-3 is our last warning shot" for coordinating to address AGI alignment. (Podcast version.) I include this talk because it's a good talk and the topic warrants discussion, though MIRI staff don't necessarily endorse this claim — and Eliezer would certainly object to any claim that something is a fire alarm for AGI.
OpenAI safety researchers including Dario Amodei, Paul Christiano, and Chris Olah depart OpenAI.
OpenAI's DALL-E uses GPT-3 for image generation, while CLIP exhibits impressive zero-shot image classification capabilities. Gwern Branwen comments in his newsletter.

The post February 2021 Newsletter appeared first on Machine Intelligence Research Institute.

February 2021 Newsletter


January 2021 Newsletter

MIRI updates

MIRI's Evan Hubinger uses a notion of optimization power to define whether AI systems are compatible with the strategy-stealing assumption.
MIRI's Abram Demski discusses debate approaches to AI safety that don't rely on factored cognition.
Evan argues that the first AGI systems are likely to be very similar to each other, and discusses implications for alignment.
Jack Clark's Import AI newsletter discusses the negative research results from our end-of-year update.
Richard Ngo shares high-quality discussion of his great introductory sequence AGI Safety from First Principles, featuring Paul Christiano, Max Daniel, Ben Garfinkel, Adam Gleave, Matthew Graves, Daniel Kokotajlo, Will MacAskill, Rohin Shah, Jaan Tallinn, and MIRI's Evan Hubinger and Buck Shlegeris.
Tom Chivers discusses the rationality community and Rationality: From AI to Zombies. (Contrary to the headline, COVID-19 receives little discussion.)

News and links

Alex Flint argues that growing the field of AI alignment researchers should be a side-effect of optimizing for "research depth", rather than functioning as a target in its own right — much as software projects shouldn't optimize for larger teams or larger codebases. Flint also comments on strategy/policy research.
Daniel Kokotajlo of the Center on Long-Term Risk argues that AGI may enable decisive strategic advantage before GDP accelerates. Cf. a pithier comment from Eliezer Yudkowsky.
TAI Safety Bibliographic Database: Jess Riedel and Angelica Deibel release a database of AI safety research, and analyze recent trends in the field.
DeepMind's AI safety team investigates optimality properties of meta-trained RNNs and the tampering problem: "How can we design agents that pursue a given objective when all feedback mechanisms for describing that objective are influenceable by the agent?"
Facebook launches Forecast, a new community prediction platform akin to Metaculus.
Effective altruists have released the microCOVID calculator, a very handy tool for assessing activities' COVID-19 infection risk. Meanwhile, Zvi Mowshowitz's weekly updates on LessWrong continue to be a good (US-centric) resource for staying up to date on COVID-19 developments such as the B117 variant.
Rethink Priorities researcher Linchuan Zhang summarizes things he's learned forecasting COVID-19 in 2020: forming good outside views is often hard; effective altruists tend to overrate superforecasters; etc.

The post January 2021 Newsletter appeared first on Machine Intelligence Research Institute.

January 2021 Newsletter


December 2020 Newsletter

MIRI COO Malo Bourgon reviews our past year and discusses our future plans in 2020 Updates and Strategy.
Our biggest update is that we've made less concrete progress than we expected on the new research we described in 2018 Update: Our New Research Directions. As a consequence, we're scaling back our work on these research directions, and looking for new angles of attack that have better odds of resulting in a solution to the alignment problem.
Other MIRI updates

A new paper from MIRI researcher Evan Hubinger: "An Overview of 11 Proposals for Building Safe Advanced AI."
A belated paper announcement from last year: Andrew Critch's "A Parametric, Resource-Bounded Generalization of Löb's Theorem, and a Robust Cooperation Criterion for Open-Source Game Theory", a result originally written up during his time at MIRI, has been published in the Journal of Symbolic Logic.
MIRI's Abram Demski introduces Learning Normativity: A Research Agenda. See also Abram's new write-up, Normativity.
Evan Hubinger clarifies inner alignment terminology.
The Survival and Flourishing Fund (SFF) has awarded MIRI $563,000 in its latest round of grants! Our enormous gratitude to SFF's grant recommenders and funders.
A Map That Reflects the Territory is a new print book set collecting the top LessWrong essays of 2018, including essays by MIRI researchers Eliezer Yudkowsky, Abram Demski, and Scott Garrabrant.
DeepMind's Rohin Shah gives his overview of Scott Garrabrant's Cartesian Frames framework.

News and links

Daniel Filan launches the AI X-Risk Research Podcast (AXRP) with episodes featuring Adam Gleave, Rohin Shah, and Andrew Critch.
DeepMind's AlphaFold represents a very large advance in protein structure prediction.
Metaculus launches Forecasting AI Progress, an open four-month tournament to predict advances in AI, with a $50,000 prize pool.
Continuing the Takeoffs Debate: Richard Ngo responds to Paul Christiano's "changing selection pressures" argument against hard takeoff.
OpenAI's Beth Barnes discusses the obfuscated arguments problem for AI safety via debate:
Previously we hoped that debate/IDA could verify any knowledge for which such human-understandable arguments exist, even if these arguments are intractably large. We hoped the debaters could strategically traverse small parts of the implicit large argument tree and thereby show that the whole tree could be trusted.
The obfuscated argument problem suggests that we may not be able to rely on debaters to find flaws in large arguments, so that we can only trust arguments when we could find flaws by recursing randomly—e.g. because the argument is small enough that we could find a single flaw if one existed, or because the argument is robust enough that it is correct unless it has many flaws.

Some AI Research Areas and Their Relevance to Existential Safety: Andrew Critch compares out-of-distribution robustness, agent foundations, multi-agent RL, preference learning, and other research areas.
Ben Hoskin releases his 2020 AI Alignment Literature Review and Charity Comparison.
Open Philanthropy summarizes its AI governance grantmaking to date.

The post December 2020 Newsletter appeared first on Machine Intelligence Research Institute.

December 2020 Newsletter


2020 Updates and Strategy

MIRI's 2020 has been a year of experimentation and adjustment. In response to the COVID-19 pandemic, we largely moved our operations to more rural areas in March, and shifted to a greater emphasis on remote work. We took the opportunity to try new work set-ups and approaches to research, and have been largely happy with the results.
At the same time, 2020 saw limited progress in the research MIRI's leadership had previously been most excited about: the new research directions we started in 2017. Given our slow progress to date, we are considering a number of possible changes to our strategy, and MIRI's research leadership is shifting much of their focus toward searching for more promising paths.

Last year, I projected that our 2020 budget would be $6.4M–$7.4M, with a point estimate of $6.8M. I now expect that our 2020 spending will be slightly above $7.4M. The increase in spending above my point estimate largely comes from expenses we incurred relocating staff and taking precautions in response to the COVID-19 pandemic.
Our budget for 2021 is fairly uncertain, given that we are more likely than usual to see high-level shifts in our strategy in the coming year. My current estimate is that our spending will fall somewhere between $6M and $7.5M, which I expect to roughly break down as follows:

I'm also happy to announce that the Survival and Flourishing Fund (SFF) has awarded MIRI $563,000 to support our research going forward, on top of support they provided earlier this year.
Given that our research program is in a transitional period, and given the strong support we have already received this year—$4.38M from Open Philanthropy, $903k from SFF, and ~$1.1M from other contributors (thank you all!)—we aren't holding a formal fundraiser this winter. Donations are still welcome and appreciated during this transition; but we'll wait to make our case to donors when our plans are more solid. For now, see our donate page if you are interested in supporting our research.
Below, I'll go into more detail on how our 2020 has gone, and on our plans for the future.

2017-Initiated Research Directions and Research Plans
In 2017, we introduced a new set of research directions, which we described and motivated more in "2018 Update: Our New Research Directions." We wrote that we were "seeking entirely new low-level foundations for optimization," "endeavoring to figure out parts of cognition that can be very transparent as cognition," and "experimenting with some specific alignment problems." In December 2019, we noted that we felt we were making "steady progress" on this research, but were disappointed with the concrete results we'd had to date.
After pushing more on these lines of research, MIRI senior staff have become more pessimistic about this approach. MIRI executive director and senior researcher Nate Soares writes:
The non-public-facing research I (Nate) was most excited about had a flavor of attempting to develop new pragmatically-feasible foundations for alignable AI, that did not rely on routing through gradient-descent-style machine learning foundations. We had various reasons to hope this could work, despite the obvious difficulties.
That project has, at this point, largely failed, in the sense that neither Eliezer nor I have sufficient hope in it for us to continue focusing our main efforts there. I'm uncertain whether it failed due to implementation failures on our part, due to the inherent difficulty of the domain, or due to flaws in the underlying theory.
Part of the reason we lost hope is a sense that we were moving too slowly, given our sense of how far off AGI may be and our sense of the difficulty of the alignment problem. The field of AI alignment is working under a deadline, such that if work is going sufficiently slowly, we're better off giving up and pivoting to new projects that have a real chance of resulting in the first AGI systems being built on alignable foundations.
We are currently in a state of regrouping, weighing our options, and searching for plans that we believe may yet have a shot at working.
Looking at the field as a whole, MIRI's research leadership remains quite pessimistic about most alignment proposals that we have seen put forward so far. That is, our update toward being more pessimistic about our recent research directions hasn't reduced our pessimism about the field of alternatives, and the next directions we undertake are not likely to resemble the directions that are popular outside of MIRI today.
MIRI sees the need for a change of course with respect to these projects. At the same time, many (including Nate) still have some hope in the theory underlying this research, and have hope that the projects may be rescued in some way, such as by discovering and correcting failures in how we approached this research. But time spent on rescue efforts trades off against finding better and more promising alignment plans.
So we're making several changes affecting staff previously focused on this work. Some are departing MIRI for different work, as we shift direction away from lines they were particularly suited for. Some are seeking to rescue the 2017-initiated lines of research. Some are pivoting to different experiments and exploration.
We are uncertain about what long-term plans we'll decide on, and are in the process of generating new possible strategies. Some (non-mutually-exclusive) possibilities include:

We may become a home to diverse research approaches aimed at developing a new path to alignment. Given our increased uncertainty about the best angle of attack, it may turn out to be valuable to house a more diverse portfolio of projects, with some level of intercommunication and cross-pollination between approaches.
We may commit to an entirely new approach after a period of exploration, if we can identify one that we believe has a real chance of ensuring positive outcomes from AGI.
We may carry forward theories and insights from our 2017-initiated research directions into future plans, in a different form.

Research Write-Ups
Although our 2017-initiated research directions have been our largest focus over the last few years, we've been running many other research programs in parallel with it.
The bulk of this work is nondisclosed-by-default as well, but it includes work we've written up publicly. (Note that as a rule, this public-facing work is unrepresentative of our research as a whole.)
From our perspective, our most interesting public work this year is Scott Garrabrant's Cartesian frames model and Vanessa Kosoy's work on infra-Bayesianism.
Cartesian frames is a new framework for thinking about agency, intended as a successor to the cybernetic agent model. Whereas the cybernetic agent model assumes as basic an agent and environment persisting across time with a defined and stable I/O channel, Cartesian frames treat these features as more derived and dependent on how one conceptually carves up physical situations.
The Cartesian Frames sequence focuses especially on finding derived, approximation-friendly versions of the notion of "subagent" (previously discussed in "Embedded Agency") and temporal sequence (a source of decision-theoretic problems in cases where agents can base their decisions on predictions or proofs about their own actions). The sequence's final post discusses these and other potential directions for future work for the field to build on.
In general, MIRI's researchers are quite interested in new conceptual frameworks like these, as research progress can often be bottlenecked on our using the wrong lenses for thinking about problems, or on our lack of a simple formalism for putting intuitions to the test.
Meanwhile, Vanessa Kosoy and Alex Appel's infra-Bayesianism is a novel framework for modeling reasoning in cases where the reasoner's hypothesis space may not include the true environment.
This framework is interesting primarily because it seems applicable to such a wide variety of problems: non-realizability, decision theory, anthropics, embedded agency, reflection, and the synthesis of induction/probability with deduction/logic. Vanessa describes infra-Bayesianism as "opening the way towards applying learning theory to many problems which previously seemed incompatible with it."
2020 also saw a large update to Scott and Abram's "Embedded Agency," with some discussions clarified and several new subsections added. Additionally, a revised version of Vanessa's "Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm," co-authored with Alex Appel, was published in the Journal of Applied Logics.
To give a picture of some of the other research areas we've been pushing on, we asked some MIRI researchers and research associates to pick out highlights from their work over the past year, with comments on their selections.
Abram Demski highlights the following write-ups:

"An Orthodox Case Against Utility Functions" — "Although in some sense this is a small technical point, it is indicative of a shift in perspective in some recent agent foundations research which I think is quite important."
"Radical Probabilism" — "Again, although one could see this as merely an explanation of the older logical induction result, I think it points at an important shift in perspective."
"Learning Normativity: A Research Agenda" — "In a sense, this research agenda clarifies the shift in perspective which the above two posts were communicating, although I haven't tied everything together yet.
"Dutch-Booking CDT: Revised Argument" — "To my eye, this is a large-ish decision theory result."

Evan Hubinger summarizes his public research from the past year:

"An Overview of 11 Proposals for Building Safe Advanced AI" — "Probably my biggest project this year, this paper is my attempt at a unified explanation of the current major leading prosaic alignment proposals. The paper includes an exploration of each proposal's pros and cons from the perspective of outer alignment, inner alignment, training competitiveness, and performance competitiveness."
"I started mentoring Adam Shimi and Mark Xu this year, helping them start spinning up work in AI safety. Concrete things that came out of this include Adam Shimi's 'Universality Unwrapped' and Mark Xu's 'Does SGD Produce Deceptive Alignment?'"
"I spent a lot of time this year thinking about AI safety via debate, which resulted in two new alternative debate proposals: 'AI Safety via Market Making' and 'Synthesizing Amplification and Debate.'"
"I spent some time looking at different alignment proposals from a computational complexity standpoint, resulting in 'Alignment Proposals and Complexity Classes' and 'Weak HCH Accesses EXP.'
"'Outer Alignment and Imitative Amplification' makes the case for why imitative amplification is outer aligned; 'Learning the Prior and Generalization' provides my perspective on Paul's new 'learning the prior' approach; and 'Clarifying Inner Alignment Terminology' revisits terminology from 'Risks from Learned Optimization.'"

Earlier this year, Buck Shlegeris (link) and Evan Hubinger (link) also appeared on the Future of Life Institute's AI Alignment Podcast. Buck also gave a talk at Stanford: "My Personal Cruxes for Working on AI Safety."
Lastly, Future of Humanity Institute researcher and MIRI research associate Stuart Armstrong summarizes his own research highlights:

"Pitfalls of Learning a Reward Function Online," working with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg — "This shows how agents can manipulate a "learning" process, the conditions that make that learning actually uninfluenceable, and some methods for turning influenceable learning processes into uninfluenceable ones."
"Model Splintering" — "Here I argue that a lot of AI safety problems can be reduced to the same problem: that of dealing with what happens when you move out of distribution from the training data. I argue that a principled way of dealing with these "model splinterings" is necessary to get safe AI, and sketch out some examples."
"Syntax, Semantics, and Symbol Grounding, Simplified" — "Here I argue that symbol grounding is a practical, necessary thing, not an abstract philosophical concept."

Process Improvements and Plans
Given the unusual circumstances brought on by the COVID-19 pandemic, in 2020 MIRI decided to run various experiments to see if we could improve our researchers' productivity while our Berkeley office was unavailable. In the process, a sizable subset of our research team has found good modifications to our work environment that we aim to maintain and expand on.
Many of our research staff who spent this year in live-work quarantine groups in relatively rural areas in response to the COVID-19 pandemic have found surprisingly large benefits from living in a quieter, lower-density area together with a number of other researchers. Coordination and research have felt faster at a meta-level, with shorter feedback cycles, more efforts on more cruxy experiments, and more resulting pivots. Our biggest such pivot has been away from our 2017-initiated research directions, as described above.
Separately, MIRI staff have been weighing the costs and benefits of possibly moving somewhere outside the Bay Area for several years—taking into account the housing crisis and other governance failures, advantages and disadvantages of the local culture, tail risks of things taking a turn for the worse in the future, and other factors.
Partly as a result of these considerations, and partly because it's easier to move when many of us have already relocated this year due to COVID-19, MIRI is considering relocating away from Berkeley. As we weigh the options, a particularly large factor in our considerations is whether our researchers expect the location, living situation, and work setup to feel good and comfortable, as we generally expect this to result in improved research progress. Increasingly, this factor is pointing us towards moving someplace new.
Many at MIRI have noticed in the past that there are certain social settings, such as small effective altruism or alignment research retreats, that seem to spark an unusually high density of unusually productive conversations. Much of the energy and vibrancy in such retreats presumably stems from their novelty and their time-boxed nature. However, we suspect this isn't the only reason these events tend to be dense and productive, and we believe that we may be able to create a space that has some of these features every day.
This year, a number of our researchers have indeed felt that our new work set-up during the pandemic has a lot of this quality. We're therefore very eager to see if we can modify MIRI as a workplace so as to keep this feature around, or further augment it.

Our year, then, has been characterized by some significant shifts in our thinking about research practices and which research directions are most promising.
Although we've been disappointed by our level of recent concrete progress toward understanding how to align AGI-grade optimization, we plan to continue capitalizing on MIRI's strong pool of talent and accumulated thinking about alignment as we look for new and better paths forward. We'll provide more updates about our new strategy as our plans solidify.
The post 2020 Updates and Strategy appeared first on Machine Intelligence Research Institute.

2020 Updates and Strategy


November 2020 Newsletter

MIRI researcher Scott Garrabrant has completed his Cartesian Frames sequence. Scott also covers the first two posts' contents in video form.

Other MIRI updates

Contrary to my previous announcement, MIRI won't be running a formal fundraiser this year, though we'll still be participating in Giving Tuesday and other matching opportunities. To donate and get information on tax-advantaged donations, employer matching, etc., see intelligence.org/donate. We'll also be doing an end-of-the-year update and retrospective in the next few weeks.
Facebook's Giving Tuesday matching event takes place this Tuesday (Dec. 1) at 5:00:00am PT.  Facebook will 100%-match the first $2M donated, something that will plausibly take only 2–3 seconds. To get 100%-matched, then, it's even more important than last year to start clicking at 4:59:59AM PT. Facebook will then 10%-match the next $50M of donations that are made. Details on optimizing your donation(s) to MIRI's Facebook Fundraiser can be found at EA Giving Tuesday, a Rethink Charity project.
Video discussion: Stuart Armstrong, Scott Garrabrant, and the AI Safety Reading Group discuss Stuart's If I Were A Well-Intentioned AI….
MIRI research associate Vanessa Kosoy raises questions about AI information hazards.
Buck Shlegeris argues that we're likely at the "hinge of history" (assuming we aren't living in a simulation).
To make it easier to find and cite old versions of MIRI papers (especially ones that aren't on arXiv), we've collected links to obsolete versions on intelligence.org/revisions.

News and links

CFAR's Anna Salamon asks: Where do (did?) stable, cooperative institutions come from?
The Center for Human-Compatible AI is accepting applications for research internships through Dec. 13.
AI Safety (virtual) Camp is accepting applications through Dec. 15.
The 4th edition of Artificial Intelligence: A Modern Approach is out, with expanded discussion of the alignment problem.
DeepMind's Rohin Shah reviews Brian Christian's new book The Alignment Problem: Machine Learning and Human Values.
Daniel Filan and Rohin Shah discuss security mindset and takeoff speeds.
Fortune profiles existential risk philanthropist Jaan Tallinn.

The post November 2020 Newsletter appeared first on Machine Intelligence Research Institute.

November 2020 Newsletter


October 2020 Newsletter

Starting today, Scott Garrabrant has begun posting Cartesian Frames, a sequence introducing a new conceptual framework Scott has found valuable for thinking about agency.
In Scott's words: Cartesian Frames are "applying reasoning like Pearl's to objects like game theory's, with a motivation like Hutter's".
Scott will be giving an online talk introducing Cartesian frames this Sunday at 12pm PT on Zoom (link). He'll also be hosting office hours on Gather.Town the next four Sundays; see here for details.
Other MIRI updates

Abram Demski discusses the problem of comparing utilities, highlighting some non-obvious implications.
In March 2020, the US Congress passed the CARES Act, which changes the tax advantages of donations to qualifying NPOs like MIRI in 2020. Changes include:

1.  A new "above the line" tax deduction: up to $300 per taxpayer ($600 for a married couple) in annual charitable contributions for people who take the standard deduction. Donations to donor-advised funds (DAFs) do not qualify for this new deduction.

2.  New charitable deduction limits: Taxpayers who itemize their deductions can deduct much greater amounts of their contributions. Individuals can elect to deduct donations up to 100% of their 2020 AGI — up from 60% previously. This higher limit also does not apply to donations to DAFs.

	As usual, consult with your tax advisor for more information.
Our fundraiser this year will start on November 29 (two days before Giving Tuesday) and finish on January 2. We're hoping that having our fundraiser straddle 2020 and 2021 will give people more flexibility given the unusual tax law changes above.
I'm happy to announce that MIRI has received a donation of $246,435 from an anonymous returning donor. Our thanks to the donor, and to Effective Giving UK for facilitating this donation!

News and links

Richard Ngo tries to provide a relatively careful and thorough version of the standard argument for worrying about AGI risk: AGI Safety from First Principles. See also Rohin Shah's summary.
The AI Alignment Podcast interviews Andrew Critch about his recent overview paper, "AI Research Considerations for Human Existential Safety."

The post October 2020 Newsletter appeared first on Machine Intelligence Research Institute.

October 2020 Newsletter


September 2020 Newsletter

Abram Demski and Scott Garrabrant have made a major update to "Embedded Agency", with new discussions of ε-exploration, Newcomblike problems, reflective oracles, logical uncertainty, Goodhart's law, and predicting rare catastrophes, among other topics.

Abram has also written an overview of what good reasoning looks in the absence of Bayesian updating: Radical Probabilism. One recurring theme:
[I]n general (i.e., without any special prior which does guarantee convergence for restricted observation models), a Bayesian relies on a realizability (aka grain-of-truth) assumption for convergence, as it does for some other nice properties. Radical probabilism demands these properties without such an assumption.
[… C]onvergence points at a notion of "objectivity" for the radical probabilist. Although the individual updates a radical probabilist makes can go all over the place, the beliefs must eventually settle down to something. The goal of reasoning is to settle down to that answer as quickly as possible.
Meanwhile, Infra-Bayesianism is a new formal framework for thinking about optimal reasoning without requiring an reasoner's true environment to be in its hypothesis space. Abram comments: "Alex Appel and Vanessa Kosoy have been working hard at 'Infra-Bayesianism', a new approach to RL which aims to make it easier (ie, possible) to prove safety-relevant theorems (and, also, a new approach to Bayesianism more generally).
Other MIRI updates

Abram Demski writes a parable on the differences between logical inductors and Bayesians: The Bayesian Tyrant.
Building on the selection vs. control distinction, Abram contrasts "mesa-search" and "mesa-control".

News and links

From OpenAI's Stiennon et al.: Learning to Summarize with Human Feedback. MIRI researcher Eliezer Yudkowsky comments:
A very rare bit of research that is directly, straight-up relevant to real alignment problems! They trained a reward function on human preferences and then measured how hard you could optimize against the trained function before the results got actually worse.
[… Y]ou can ask for results as good as the best 99th percentile of rated stuff in the training data (a la Jessica Taylor's quantilization idea).  Ask for things the trained reward function rates as "better" than that, and it starts to find "loopholes" as seen from outside the system; places where the trained reward function poorly matches your real preferences, instead of places where your real preferences would rate high reward.

Chi Nguyen writes up an introduction to Paul Christiano's iterated amplification research agenda that seeks to be the first such resource that is "both easy to understand and [gives] a complete picture". The post includes inline comments by Christiano.
Forecasters share visualizations of their AI timelines on LessWrong.

The post September 2020 Newsletter appeared first on Machine Intelligence Research Institute.

September 2020 Newsletter


August 2020 Newsletter

MIRI updates

Three questions from MIRI's Abram Demski: What does it mean to apply decision theory?, How "honest" is GPT-3?, and How should AI debate be judged?
A transcript from MIRI researcher Scott Garrabrant: What Would I Do? Self-Prediction in Simple Algorithms.
MIRI researcher Buck Shlegeris reviews the debate on what the history of nuclear weapons implies about humanity's ability to coordinate.
From MIRI's Evan Hubinger: Learning the Prior and Generalization and Alignment Proposals and Complexity Classes.
Rafael Harth's Inner Alignment: Explain Like I'm 12 Edition summarizes the concepts and takeaways from "Risks from Learned Optimization".
Issa Rice reviews discussion to date on MIRI's research focus, "To what extent is it possible to have a precise theory of rationality?", and the relationship between deconfusion research and safety outcomes. (Plus a short reply.)
"Pitfalls of Learning a Reward Function Online" (IJCAI paper, LW summary): FHI researcher and MIRI research associate Stuart Armstrong, with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg, explore ways to discourage agents from manipulating their reward signal to be easier to optimize.

News and links

From Paul Christiano: Learning the Prior and Better Priors as a Safety Problem.
From Victoria Krakovna: Tradeoff Between Desirable Properties for Baseline Choices in Impact Measures.
Ben Pace summarizes Christiano's "What Failure Looks Like" post and the resultant discussion.
Kaj Sotala collects recent examples of experiences from people working with GPT-3.

The post August 2020 Newsletter appeared first on Machine Intelligence Research Institute.

August 2020 Newsletter


July 2020 Newsletter

After completing a study fellowship at MIRI that he began in late 2019, Blake Jones is joining the MIRI research team full-time! Blake joins MIRI after a long career working on low-level software systems such as the Solaris operating system and the Oracle database.
Other MIRI updates

MIRI researcher Evan Hubinger goes on the FLI podcast (transcript/discussion, audio) to discuss "inner alignment, outer alignment, and proposals for building safe advanced AI".
A revised version of Vanessa Kosoy's "Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm," co-authored with Alex Appel, has been accepted to the Journal of Applied Logics.
From MIRI researcher Abram Demski: Dutch-Booking CDT: Revised Argument argues that "causal" theories (ones using counterfactuals to evaluate expected value) must behave the same as theories using conditional probabilities. Relating HCH and Logical Induction discusses amplification in the context of reflective oracles. And Radical Probabilism reviews the surprising gap between Dutch-book arguments and Bayes' rule.

News and links

Alex Flint summarizes the Center for Human-Compatible AI's assistance games research program.
CHAI's Andrew Critch and MILA's David Krueger release "AI Research Considerations for Human Existential Safety (ARCHES)", a review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.
OpenAI's Danny Hernandez and Tom Brown present evidence that "for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency".
DeepMind's Victoria Krakovna shares her takeaways from the COVID-19 pandemic for slow-takeoff scenarios.
AI Impacts' Daniel Kokotajlo discusses possible changes the world might undergo before reaching AGI.
80,000 Hours describes careers they view as promising but haven't written up as priority career paths, including information security (previously discussed here).

The post July 2020 Newsletter appeared first on Machine Intelligence Research Institute.

July 2020 Newsletter