 Computing has changed how people communicate. The transmission of news, messages, and ideas is instant. Anyone’s voice can be heard. In fact, access to digital communication technologies such as the Internet is so fundamental to daily life that their disruption by government is condemned by the United Nations Human Rights Council . But while the technology to distribute our ideas has grown in leaps and bounds, the interfaces have remained largely the same.

 Parallel to the development of the internet, researchers like Alan Kay and Douglas Engelbart worked to build technology that would empower individuals and enhance cognition. Kay imagined the Dynabook  in the hands of children across the world. Engelbart, while best remembered for his “mother of all demos,” was more interested in the ability of computation to augment human intellect . Neal Stephenson wrote speculative fiction that imagined interactive paper that could display videos and interfaces, and books that could teach and respond to their readers .

 More recent designs (though still historical by personal computing standards) point to a future where computers are connected and assist people in decision-making and communicating using rich graphics and interactive user interfaces . While some technologies have seen mainstream adoption, such as Hypertext , unfortunately, many others have not. The most popular publishing platforms, for example WordPress and Medium, choose to prioritize social features and ease-of-use while limiting the ability for authors to communicate using the dynamic features of the web.

 In the spirit of previous computer-assisted cognition technologies, a new type of computational communication medium has emerged that leverages active reading techniques to make ideas more accessible to a broad range of people. These interactive articles build on a long history, from Plato  to PHeT  to explorable explanations . They have been shown to be more engaging, can help improve recall and learning, and attract broad readership and acclaim,For example, some of *The New York Times*  and *The Washington Post’s*  most read articles are interactive stories. yet we do not know that much about them.

 In this work, for the the first time, we connect the dots between interactive articles such as those featured in this journal and publications like *The New York Times* and the techniques, theories, and empirical evaluations put forth by academic researchers across the fields of education, human-computer interaction, information visualization, and digital journalism. We show how digital designers are operationalizing these ideas to create interactive articles that help boost learning and engagement for their readers compared to static alternatives.

 Today there is a growing excitement around the use of interactive articles for communication since they offer unique capabilities to help people learn and engage with complex ideas that traditional media lacks. After describing the affordances of interactive articles, we provide critical reflections from our own experience with open-source, interactive publishing at scale. We conclude with discussing practical challenges and open research directions for authoring, designing, and publishing interactive articles.

 This style of communication — and the platforms which support it — are still in their infancy. When choosing where to publish this work, we wanted the medium to reflect the message. Journals like *Distill* are not only pushing the boundaries of machine learning research but also offer a space to put forth new interfaces for dissemination. This work ties together the theory and practice of authoring and publishing interactive articles. It demonstrates the power that the medium has for providing new representations and interactions to make systems and ideas more accessible to broad audiences.

Interactive Articles: Theory & Practice
---------------------------------------

 Interactive articles draw from and connect many types of media, from static text and images to movies and animations. But in contrast to these existing forms, they also leverage interaction techniques such as details-on demand, belief elicitation, play, and models and simulations to enhance communication.

 While the space of possible designs is far too broad to be solved with one-size-fits-all guidelines, by connecting the techniques used in these articles back to underlying theories presented across disparate fields of research we provide a missing foundation for designers to use when considering the broad space of interactions that could be added to a born-digital article.

 We draw from a corpus of over sixty interactive articles to highlight the breadth of techniques available and analyze how their authors took advantage of a digital medium to improve the reading experience along one or more dimensions, for example, by reducing the overall cognitive load, instilling positive affect, or improving information recall.

 Because diverse communities create interactive content, this medium goes by many different names and has not yet settled on a standardized format nor definition.However, one is taking shape. Researchers have proposed artifacts such as explorable multiverse analyses , explainables , and exploranations  to more effectively disseminate their work, communicate their results to the public, and remove research debt . In newsrooms, data journalists, developers, and designers work together to make complex news and investigative reporting clear and engaging using interactive stories . Educators use interactive textbooks as an alternative learning format to give students hands-on experience with learning material .

 Besides these groups, others such as academics, game developers, web developers, and designers blend editorial, design, and programming skills to create and publish explorable explanations , interactive fiction , interactive non-fiction , active essays , and interactive games . While these all slightly differ in their technical approach and target audience, they all largely leverage the interactivity of the modern web.

 We focus on five unique affordances of interactive articles, listed below. In-line videos and example interactive graphics are presented alongside this discussion to demonstrate specific techniques.

### Connecting People and Data

 As visual designers are well aware, and as journalism researchers have confirmed empirically , an audience which finds content to be aesthetically pleasing is more likely to have a positive attitude towards it. This in turn means people will spend more time engaging with content and ultimately lead to improved learning outcomes. While engagement itself may not be an end goal of most research communications, the ability to influence both audience attitude and the amount of time that is spent is a useful lever to improve learning: we know from education research that both time spent  and emotion  are predictive of learning outcomes.

 Animations can also be used to improve engagement . While there is debate amongst researchers if animations in general are able to more effectively convey the same information compared to a well designed static graphic , animation has been shown to be effective specifically for communicating state transitions , uncertainty , causality , and constructing narratives . A classic example of this is Muybridge’s motion study  that can be seen in [3](#horse): while the series of still images may be more effective for answering specific questions like, “Does a horse lift all four of its feet off the ground when it runs?” watching the animation in slow motion gives the viewer a much more visceral sense of how it runs. A more modern example can be found in OpenAI’s reporting on their hide-and-seek agents . The animations here instantly give the viewer a sense of how the agents are operating in their environment.

 Passively, animation can be used to add drama to a graphic displaying important information, but which readers may otherwise find dry. Scientific data which is inherently time varying may be shown using an animation to connect viewers more closely with the original data, as compared to seeing an abstracted static view. For example, Ed Hawkins designed “Climate Spirals,” which shows the average global temperature change over time . This presentation of the data resonated with a large public audience, so much so that it was displayed at the opening ceremony at the 2016 Rio Olympics. In fact, many other climate change visualizations of this same dataset use animation to build suspense and highlight the recent spike in global temperatures .

 By adding variation over time, authors have access to a new dimension to encode information and an even wider design space to work in. Consider the animated graphic in *The New York Times* story “Extensive Data Shows Punishing Reach of Racism for Black Boys,” which shows economic outcomes for 10,000 men who grew up in rich families . While there are many ways in which the same data could have been communicated more succinctly using a static visualization , by utilizing animation, it became possible for the authors to design a unit visualization in which each data point shown represented an individual, reminding readers that the data in this story was about real peoples’ lives.

 Unit visualizations have also been used to evoke empathy in readers in other works covering grim topics such as gun deaths  and soldier deaths in war . Using person-shaped glyphs (as opposed to abstract symbols like circles or squares) has been shown not to produce additional empathic responses , but including actual photographs of people helps readers gain interest in, remember , and communicate complex phenomena  using visualizations. Correll argues that much of the power of visualization comes from abstraction, but quantization stymies empathy . He instead suggests anthropomorphizing data, borrowing journalistic and rhetoric techniques to create novel designs or interventions to foster empathy in readers when viewing visualizations .

 Regarding the format of interactive articles, an ongoing debate within the data journalism community has been whether articles which utilize scroll-based graphics (scrollytelling) are more effective than those which use step-based graphics (slideshows). McKenna et al.  found that their study participants largely preferred content to be displayed with a step- or scroll-based navigation as opposed to traditional static articles, but did not find a significant difference in engagement between the two layouts. In related work, Zhi et al. found that performance on comprehension tasks was better in slideshow layouts than in vertical scroll-based layouts . Both studies focused on people using desktop (rather than mobile) devices. More work is needed to evaluate the effectiveness of various layouts on mobile devices, however the interviews conducted by MckEnna et al. suggest that additional features, such as supporting navigation through swipe gestures, may be necessary to facilitate the mobile reading experience.

 The use of games to convey information has been explored in the domains of journalism  and education . Designers of newsgames use them to help readers build empathy with their subject, for example in *The Financial Times’s* “Uber Game” , and explain complex systems consisting of multiple parts, for example in *Wired’s* “Cutthroat Capitalism: The Game” . In educational settings the use of games has been shown to motivate students while maintaining or improving learning outcomes .

 As text moves away from author-guided narratives towards more reader-driven ones , the reading experience becomes closer to that of playing a game. For example, the critically acclaimed explorable explanation “Parable of the Polygons” puts play at the center of the story, letting a reader manually run an algorithm that is later simulated in the article to demonstrate how a population of people with slight personal biases against diversity leads to social segregation .

### Making Systems Playful

 Interactive articles utilize an underlying computational infrastructure, allowing authors editorial control over the computational processes happening on a page. This access to computation allows interactive articles to engage readers in an experience they could not have with traditional media. For example, in “Drawing Dynamic Visualizations”, Victor demonstrates how an interactive visualization can allow readers to build an intuition about the behavior of a system, leading to a fundamentally different understanding of an underlying system compared to looking at a set of static equations . These articles leverage active learning and reading, combined with critical thinking  to help diverse sets of people learn and explore using sandboxed models and simulations .

 Complex systems often require extensive setup to allow for proper study: conducting scientific experiments, training machine learning models, modeling social phenomenon, digesting advanced mathematics, and researching recent political events, all require the configuration of sophisticated software packages before a user can interact with a system at all, even just to tweak a single parameter. This barrier to entry can deter people from engaging with complex topics, or explicitly prevent people who do not have the necessary resources, for example, computer hardware for intense machine learning tasks. Interactive articles drastically lower these barriers.

 Science that utilizes physical and computational experiments requires systematically controlling and changing parameters to observe their effect on the modeled system. In research, dissemination is typically done through static documents, where various figures show and compare the effect of varying particular parameters. However, efforts have been made to leverage interactivity in academic publishing, summarized in . Reimagining the research paper with interactive graphics , as exploranations , or as explorable multiverse analyses , gives readers control over the reporting of the research findings and shows great promise in helping readers both digest new ideas and learn about existing fields that are built upon piles of research debt .

 Beyond reporting statistics, interactive articles are extremely powerful when the studied systems can be modeled or simulated in real-time with interactive parameters without setup, e.g., in-browser sandboxes. Consider the example in [4](#simulation-vis) of a Boids simulation that models how birds flock together. Complex systems such as these have many different parameters that change the resulting simulation. These sandbox simulations allow readers to play with parameters to see their effect without worrying about technical overhead or other experimental consequences.

 This is a standout design pattern within interactive articles, and many examples exist ranging in complexity. “How You Will Die” visually simulates the average life expectancy of different groups of people, where a reader can choose the gender, race, and age of a person . “On Particle Physics” allows readers to experiment with accelerating different particles through electric and magnetic fields to build intuition behind electromagnetism foundations such as the Lorentz force and Maxwell’s equations — the experiments backing these simulations cannot be done without multi-million dollar machinery . “Should Prison Sentences Be Based On Crimes That Haven’t Been Committed Yet?” shows the outcome of calculating risk assessments for recidivism where readers adjust the thresholds for determining who gets parole .

 The dissemination of modern machine learning techniques has been bolstered by interactive models and simulations. Three articles, “How to Use t-SNE Effectively” , “The Beginner’s Guide to Dimensionality Reduction” , and “Understanding UMAP”  show the effect that hyperparameters and different dimensionality reduction techniques have on creating low dimensional embeddings of high-dimensional data. A popular approach is to demonstrate how machine learning models work with in-browser models , for example, letting readers use their own video camera as input to an image classification model  or handwriting as input to a stroke prediction model . Other examples are aimed at technical readers who wish to learn about specific concepts within deep learning. Here, interfaces allow readers to choose model hyperparameters, datasets, and training procedures that, once selected, visualize the training process and model internals to inspect the effect of varying the model configuration .

 Interactive articles commonly communicate a single idea or concept using multiple representations. The same information represented in different forms can have different impact. For example, in mathematics often a single object has both an algebraic and a geometric representation. A clear example of this is the definition of a circle . Both are useful, inform one another, and lead to different ways of thinking. Examples of interactive articles that demonstrate this include various media publications’ political election coverage that break down the same outcome in multiple ways, for example, by voter demographics, geographical location, and historical perspective .

 The Multimedia Principle states that people learn better from words and pictures rather than words or pictures alone , as people can process information through both a visual channel and auditory channel simultaneously. Popular video creators such as 3Blue1Brown  and Primer  exemplify these principles by using rich animation and simultaneous narration to break down complex topics. These videos additionally take advantage of the Redundancy Principle by including complementary information in the narration and in the graphics rather than repeating the same information in both channels .

 While these videos are praised for their approachability and rich exposition, they are not interactive. One radical extension from traditional video content is also incorporating user input into the video while narration plays. A series of these interactive videos on “Visualizing Quaternions” lets a reader listen to narration of a live animation on screen, but at any time the viewer can take control of the video and manipulate the animation and graphics while simultaneously listening to the narration .

 Utilizing multiple representations allows a reader to see different abstractions of a single idea. Once these are familiar and known, an author can build interfaces from multiple representations and let readers interact with them simultaneously, ultimately leading to interactive experiences that demonstrate the power of computational communication mediums. Next, we discuss such experiences where interactive articles have transformed communication and learning by making live models and simulations of complex systems and phenomena accessible.

### Prompting Self-Reflection

 Asking a student to reflect on material that they are studying and explain it back to themselves — a learning technique called self-explanation — is known to have a positive impact on learning outcomes . By generating explanations and refining them as new information is obtained, it is hypothesized that a student will be more engaged with the processes which they are studying . When writing for an interactive environment, components can be included which prompt readers to make a prediction or reflection about the material and cause them to engage in self-explanation .

 While these prompts may take the form of text entry or other standard input widgets, one of the most prominent examples of this technique used in practice comes from *The New York Times* “You Draw It” visualizations . In these visualizations, readers are prompted to complete a trendline on a chart, causing them to generate an explanation based on their current beliefs for why they think the trend may move in a certain direction. Only after readers make their prediction are they shown the actual data. Kim et al. showed that using visualizations as a prompt is an effective way to encourage readers to engage in self explanation and improve their recall of the information . [5](#you-draw-it) shows one these visualizations for CO₂ emissions from burning fossil fuels. After clicking and dragging to guess the trend, your guess will be compared against the actual data.

 In the case of “You Draw It,” readers were also shown the predictions that others made, adding a social comparison element to the experience. This additional social information was not shown to necessarily be effective for improving recall . However, one might hypothesize that this social aspect may have other benefits such as improving reader engagement, due to the popularity of recent visual stories using this technique, for example in *The Pudding’s* “Gyllenhaal Experiment”  and *Quartz’s* “How do you draw a circle?” .

 Prompting readers to remember previously presented material, for example through the use of quizzes, can be an effective way to improve their ability to recall it in the future . This result from cognitive psychology, known as the testing effect , can be utilized by authors writing for an interactive medium . While testing may call to mind stressful educational experiences for many, quizzes included in web articles can be low stakes: there is no need to record the results or grade readers. The effect is enhanced if feedback is given to the quiz-takers, for example by providing the correct answer after the user has recorded their response .

 The benefits of the testing effect can be further enhanced if the testing is repeated over a period of time , assuming readers are willing to participate in the process. The idea of spaced repetition has been a popular foundation for memory building applications, for example in the Anki flash card system. More recently, authors have experimented with building spaced repetition directly into their web-based writing , giving motivated readers the opportunity to easily opt-in to a repeated testing program over the relevant material.

### Personalizing Reading

 Content personalization — automatically modifying text and multimedia based on a reader’s individual features or input (e.g., demographics or location) — is a technique that has been shown to increase engagement and learning within readers  and support behavioral change . The PersaLog system gives developers tools to build personalized content and presents guidelines for personalization based on user research from practicing journalists . Other work has shown that “personalized spatial analogies,” presenting distance measurements in regions where readers are geographically familiar with, help people more concretely understand new distance measurements within news stories .

 Personalization alone has also been used as the standout feature of multiple interactive articles. Both “How Much Hotter Is Your Hometown Than When You Were Born?”  and “Human Terrain”  use a reader’s location to drive stories relating to climate change and population densities respectively. Other examples ask for explicit reader input, such as a story that visualizes a reader’s net worth to challenge a reader’s assumptions if they are wealthy or not (relative to the greater population) , or predicting a reader’s political party affiliation . Another example is the interactive scatterplot featured in “Find Out If Your Job Will Be Automated” . In this visualization, professions are plotted to determine their likelihood of being automated against their average annual wage. The article encourages readers to use the search bar to type in their own profession to highlight it against the others.

 An interactive medium has the potential to offer readers an experience other than static, linear text. Non-linear stories, where a reader can choose their own path through the content, have the potential to provide a more personalized experience and focus on areas of user interest . For example, the *BBC* has used this technique in both online articles  and in a recent episode of “Click” , a technology focused news television program. Non-linear stories present challenges for authors, as they must consider the myriad possible paths through the content, and consider the different possible experiences that the audience would have when pursuing different branches.

 Another technique interactive articles often use is segmenting content into small pieces to be read in-between or alongside other graphics. While we have already discussed cognitive load theory, the Segmenting Theory, the idea that complex lessons are broken into smaller, bit-sized parts , also supports personalization within interactive articles. Providing a reader the ability to play, pause, and scrub content allows the reader to move at their own speed, comprehending the information at a speed that works best for them. Segmenting also engages a reader’s essential processing without overloading their cognitive system .

 Multiple studies have been conducted showing that learners perform better when information is segmented, whether it be only within an animation  or within an interface with textual descriptions . One excellent example of using segmentation and animation to personalize content delivery is “A Visual Introduction to Machine Learning,” which introduces fundamental concepts within machine learning in bite-sized pieces, while transforming a single dataset into a trained machine learning model . Extending this idea, in “Quantum Country,” an interactive textbook covering quantum computing, the authors implemented a user account system, allowing readers to save their position in the text and consume the content at their own pace . This book further utilizes the interactive medium by utilizing spaced repetition that helps improve recall.

### Reducing Cognitive Load

 Authors must calibrate the detail at which to discuss ideas and content to their readers expertise and interest to not overload them. When topics become multifaceted and complex, a balance must be struck between a high-level overview of a topic and its lower-level details. One interaction technique used to prevent a cognitive overload within a reader is “details-on-demand.”

 Details-on-demand has become an ubiquitous design pattern. For example, modern operating systems offer to fetch dictionary definitions when a word is highlighted. When applied to visualization, this technique allows users to select parts of a dataset to be shown in more detail while maintaining a broad overview. This is particularly useful when a change of view is not required, so that users can inspect elements of interest on a point-by-point basis in the context of the whole . Below we highlight areas where details-on-demand has been successfully applied to reduce the amount of information present within an interface at once.

#### Data Visualization

 Details-on-demand is core to information visualization, and concludes the seminal Visual Information-Seeking Mantra: *“Overview first, zoom and filter, then details-on-demand”* . Successful visualizations not only provide the base representations and techniques for these three steps, but also bridge the gaps between them . In practice, the solidified standard for details-on-demand in data visualization manifests as a tooltip, typically summoned on a cursor mouseover, that presents extra information in an overlay. Given that datasets often contain multiple attributes, tooltips can show the other attributes that are not currently encoded visually , for example, the map in [6](#details-vis) that shows where different types of birdsongs where recorded and what they sound like.

#### Illustration

 Details-on-demand is also used in illustrations, interactive textbooks, and museum exhibits, where highlighted segments of a figure can be selected to display additional information about the particular segment. For example, in “How does the eye work?” readers can select segments of an anatomical diagram of the human eye to learn more about specific regions, e.g., rods and cones . Another example is “Earth Primer,” an interactive textbook on tablets that allows readers to inspect the Earth’s interior, surface, and biomes . Each illustration contains segments the reader can tap to learn and explore in depth. [7](#details-illustration) demonstrates this by pointing out specific regions in machine-generated imagery  to help people spot fake images.

#### Mathematical Notation

 Formal mathematics, a historically static medium, can benefit from details-on-demand, for example, to elucidate a reader with intuition about a particular algebraic term, present a geometric interpretation of an equation, or to help a reader retain high-level context while digesting technical details.See this [list of examples](https://github.com/fredhohman/awesome-mathematical-notation-design) that experiment with applying new design techniques to various mathematical notation. For example, in “Why Momentum Really Works,” equation layout is done using Gestalt principles plus annotation to help a reader easily identify specific terms. In “Colorized Math Equations,” the Fourier transform equation is presented in both mathematical notation and plain text, but the two are linked through a mouseover that highlights which term in the equation corresponds to which word in the text . Another example that visualizes mathematics and computation is the “Image Kernels” tutorial where a reader can mouseover a real image and observe the effect and exact computation for applying a filter over the image .

 Instead of writing down long arithmetic sums, the interface allows readers to quickly see the summation operation’s terms and output. In [8](#details-math), one of Maxwell’s equations is shown. Click the two buttons to reveal, or remind yourself, what each notation mark and variable represent.

#### Text

 While not as pervasive, text documents and other long-form textual mediums have also experimented with letting readers choose a variable level of detail to read. This idea was explored as early as the 1960s in StretchText, a hypertext feature that allows a reader to reveal a more descriptive or exhaustive explanation of something by expanding or contracting the content in place . The idea has resurfaced in more recent examples, including “On Variable Level-of-detail Documents” , a PhD thesis turned interactive article , and the call for proposals of *The Parametric Press* . One challenge that has limited this technique’s adoption is the burden it places on authors to write multiple versions of their content. For example, drag the slider in [9](#details-text) to read descriptions of the Universal Approximation Theorem in increasing levels of detail. For other examples of details-on-demand for text, such as application in code documentation, see this small collection of examples .

#### Previewing Content

 Details-on-demand can also be used as a method for previewing content without committing to another interaction or change of view. For example, when hovering over a hyperlink on Wikipedia, a preview card is shown that can contain an image and brief description; this gives readers a quick preview of the topic without clicking through and loading a new page . This idea is also not new: work from human-computer interaction explored fluid links  within hypertext that present information about a particular topic in a location that does not obscure the source material. Both older and modern preview techniques use perceptually-based animation and simple tooltips to ensure their interactions are natural and lightweight feeling to readers .

Challenges for Authoring Interactives
-------------------------------------

*If interactive articles provide clear benefits over other mediums for communicating complex ideas, then why aren’t they more prevalent?*

 Unfortunately, creating interactive articles today is difficult. Domain-specific diagrams, the main attraction of many interactive articles, must be individually designed and implemented, often from scratch. Interactions need to be intuitive and performant to achieve a nice reading experience. Needless to say, the text must also be well-written, and, ideally, seamlessly integrated with the graphics.

 The act of creating a successful interactive article is closer to building a website than writing a blog post, often taking significantly more time and effort than a static article, or even an academic publication.As a proxy, see the number of commits on an [example *Distill* article](https://github.com/distillpub/post--building-blocks). Most interactive articles are created using general purpose web-development frameworks which, while expressive, can be difficult to work with for authors who are not also web developers. Even for expert web developers, current tools offer lower levels of abstraction than may be desired to prototype and iterate on designs.

 While there are some tools that help with alleviating this problem , they are relatively immature and mainly help with reducing the necessary programming tedium. Tools like Idyll  can help authors start writing quickly and even enable rapid iteration through various designs (for example, letting an author quickly compare between sequencing content using a “scroller” or “stepper” based layout). However, Idyll does not offer any design guidance, help authors think through where interactivity would be most effectively applied, nor highlight how their content could be improved to increase its readability and memorability. For example, Idyll encodes no knowledge of the positive impact of self-explanation, instead it requires authors to be familiar with this research and how to operationalize it.

 To design an interactive article successfully requires a diverse set of editorial, design, and programming skills. While some individuals are able to author these articles on their own, many interactive articles are created by a collective team consisting of multiple members with specialized skills, for example, data analysts, scripters, editors, journalists, graphic designers, and typesetters, as outlined in . The current generation of authoring tools do not acknowledge this collaboration. For example, to edit only the text of *this article* requires one to clone its source code using git, install project-specific dependencies using a terminal, and be comfortable editing HTML files. All of this complexity is incidental to task of editing text.

 Publishing to the web brings its own challenges: while interactive articles are available to anyone with a browser, they are burdened by rapidly changing web technologies that could break interactive content after just a few years. For this reason, easy and accessible interactive article archival is important for authors to know their work can be confidently preserved indefinitely to support continued readership.This challenge has been [pointed out](https://twitter.com/redblobgames/status/1168520452634865665) by the community. Authoring interactive articles also requires designing for a diverse set of devices, for example, ensuring bespoke content can be adapted for desktop and mobile screen sizes with varying connection speeds, since accessing interactive content demands more bandwidth.

 There are other non-technical limitations for publishing interactive articles. For example, in non-journalism domains, there is a mis-aligned incentive structure for authoring and publishing interactive content: why should a researcher spend time on an “extra” interactive exposition of their work when they could instead publish more papers, a metric by which their career depends on? While different groups of people seek to maximize their work’s impact, legitimizing interactive artifacts requires buy-in from a collective of communities.

 Making interactive articles accessible to people with disabilities is an open challenge. The dynamic medium exacerbates this problem compared to traditional static writing, especially when articles combine multiple formats like audio, video, and text. Therefore, ensuring interactive articles are accessible to everyone will require alternative modes of presenting content (e.g. text-to-speech, video captioning, data physicalization, data sonification) and careful interaction design.

 It is also important to remember that not everything needs to be interactive. Authors should consider the audience and context of their work when deciding if use of interactivity would be valuable. In the worst case, interactivity may be distracting to readers or the functionality may go unused, the author having wasted their time implementing it. However, even in a domain where the potential communication improvement is incremental,In reality, multimedia studies show large effect sizes for improvement of transfer learning in many cases, see Chapter 12 of. at scale (e.g., delivering via the web), interactive articles can still [have impact](https://twitter.com/michael_nielsen/status/1031256363458916352?lang=en) .

Critical Reflections
--------------------

 We write this article not as media theorists, but as practitioners, researchers, and tool builders. While it has never been easier for writers to share their ideas online, current publishing tools largely support only static authoring and do not take full advantage of the fact that the web is a dynamic medium. We want that to change, and we are not alone. Others from the explorable explanations community have identified design patterns that help share complex ideas through play .

 To explore these ideas further, two of this work’s authors created *The Parametric Press:* an annually published digital magazine that showcases the expository power that interactive dynamic media can have when effectively combined . In late 2018, we invited writers to respond to a call for proposals for our first issue focusing on exploring scientific and technological phenomena that stand to shape society at large. We sought to cover topics that would benefit from using the interactive or otherwise dynamic capabilities of the web. Given the challenges of authoring interactive articles, we did not ask authors to submit fully developed pieces. Instead, we accepted idea submissions, and collaborated with the authors over the course of four months to develop the issue, offering technical, design, and editorial assistance collectively to the authors that lacked experience in one of these areas. For example, we helped a writer implement visualizations, a student frame a cohesive narrative, and a scientist recap history and disseminate to the public. Multiple views from one article are shown in [10](#parametric).

 We see *The Parametric Press* as a crucial connection between the often distinct worlds of research and practice. The project serves as a platform through which to operationalize the theories put forth by education, journalism, and HCI researchers. Tools like Idyll which are designed in a research setting need to be validated and tested to ensure that they are of practical use; *The Parametric Press* facilitates this by allowing us to study its use in a real-world setting, by authors who are personally motivated to complete their task of constructing a high-quality interactive article, and only have secondary concerns and care about the tooling being used, if at all.

 Through *The Parametric Press*, we saw the many challenges of authoring, designing, and publishing first hand, dually as researchers and practitioners. [2](#research-x-practice-table) summarizes interactive communication opportunities from both research and practice.

 As researchers we can treat the project as a series of case studies, where we were observers of the motivation and workflows which were used to craft the stories, from their initial conception to their publication. Motivation to contribute to the project varied by author. Where some authors had personal investment in an issue or dataset they wanted to highlight and raise awareness to broadly, others were drawn towards the medium, recognizing its potential but not having the expertise or support to communicate interactively. We also observed how research software packages like Apparatus , Idyll , and D3  fit into the production of interactive articles, and how authors must combine these disparate tools to create an engaging experience for readers. In one article, “On Particle Physics,” an author combined two tools in a way that allowed him to create and embed dynamic graphics directly into his article without writing any code beyond basic markup. One of the creators of Apparatus had not considered this type of integration before, and upon seeing the finished article [commented](https://twitter.com/qualmist/status/1128157840672051200?s=20), *“That’s fantastic! Reading that article, I had no idea that Apparatus was used. This is a very exciting proof-of-concept for unconventional explorable-explanation workflows.”*

 We were able to provide editorial guidance to the authors drawing on our knowledge of empirical studies done in the multimedia learning and information visualization communities to recommend graphical structures and page layouts, helping each article’s message be communicated most effectively. One of the most exciting outcomes of the project is that we saw authors develop interactive communication skills like any other skill: through continued practice, feedback, and iteration. We also observed the challenges that are inherent in publishing dynamic content on the web and identified the need for improved tooling in this area, specifically around the archiving of interactive articles. Will an article’s code still run a year from now? Ten years from now? To address interactive content archival, we set up a system to publish a digital archive of all of our articles at the time that they are first published to the site. At the top of each article on *The Parametric Press* is an archive link that allows readers to download a WARC (Web ARChive) file that can “played back” without requiring any web infrastructure. While our first iteration of the project relied on ad-hoc solutions to these problems, we hope to show how digital works such as ours can be published confidently knowing that they will be preserved indefinitely.

 As practitioners we pushed the boundaries of the current generation of tools designed to support the creation of interactive articles on the web. We found bugs and limitations in Idyll, a tool which was originally designed to support the creation of one-off articles that we used as a content management system to power an entire magazine issue. We were forced to write patches and plugins to work around the limitations and achieve our desired publication.Many of these patches have since been merged to Idyll itself. This is the power of modular open-source tooling in action. We were also forced to craft designs under a more realistic set of constraints than academics usually deal with: when creating a visualization it is not enough to choose the most effective visual encodings, the graphics also had to be aesthetically appealing, adhere to a house style, have minimal impact on page load time and runtime performance, be legible on both mobile and desktop devices, and not be overly burdensome to implement. Any extra hour spent implementing one graphic was an hour that was not spent improving some other part of the issue, such as the clarity of the text, or the overall site design.

 There are relatively few outlets that have the skills, technology, and desire to publish interactive articles. From its inception, one of the objectives of *The Parametric Press* is to showcase the new forms of media and publishing that are possible with tools available today, and inspire others to create their own dynamic writings. For example, Omar Shehata, the authors of *The Parametric Press* article [“Unraveling the JPEG,”](https://parametric.press/issue-01/unraveling-the-jpeg/) told us he had wanted to write this interactive article for years yet never had the opportunity, support, or incentive to create it. His article drew wide interest and critical acclaim.

 We also wanted to take the opportunity as an independent publication to serve as a concrete example for others to follow, to represent a set of best practices for publishing interactive content. To that end, we made available all of the software that runs the site, including reusable components, custom data visualizations, and the publishing engine itself.

Looking Forward
---------------

 A diverse community has emerged to meet these challenges, exploring and experimenting with what interactive articles could be. The [Explorable Explanations community](https://explorabl.es/) is a “disorganized ‘movement’ of artists, coders & educators who want to reunite play and learning.” Their online hub contains 170+ interactive articles on topics ranging from art, natural sciences, social sciences, journalism, and civics. The curious can also find tools, tutorials, and meta-discussion around learning, play, and representations. Explorables also hosted a mixed in-person and [online Jam](https://explorabl.es/jam/): a community-based sprint focused on creating new explorable explanations. [11](#jam) highlights a subset of the interactive articles created during the Jam.

 Many interactive articles are self-published due to a lack of platforms that support interactive publishing. Creating more outlets that allow authors to publish interactive content will help promote their development and legitimization. The few existing examples, including newer journals such as *Distill*, academic workshops like VISxAI , open-source publications like *The Parametric Press* , and live programming notebooks like Observable  help, but currently target a narrow group of authors, namely those who have programming skills. Such platforms should also provide clear paths to submission, quality and editorial standards, and authoring guidelines. For example, news outlets have clear instructions for pitching written pieces, yet this is under-developed for interactive articles. Lastly, there is little funding available to support the development of interactive articles and the tools that support them. Researchers do not receive grants to communicate their work, and practitioners outside of the largest news outlets are not able to afford the time and implementation investment. Providing more funding for enabling interactive articles incentivizes their creation and can contribute to a culture where readers expect digital communications to better utilize the dynamic medium.

 We have already discussed the breadth of skills required to author an interactive article. Can we help lower the barrier to entry? While there have been great, practical strides in this direction , there is still opportunity for creating tools to design, develop, and evaluate interactive articles in the wild. Specific features should include supporting mobile-friendly adaptations of interactive graphics (for example ), creating content for different platforms besides just the web, and tools that allow people to create interactive content without code.

 The usefulness of interactive articles is predicated on the assumption that these interactive articles actually facilitate communication and learning. There is limited empirical evaluation of the effectiveness of interactive articles. The problem is exacerbated by the fact that large publishers are unwilling to share internal metrics, and laboratory studies may not generalize to real world reading trends. *The New York Times* provided one of the few available data points, stating that only a fraction of readers interact with non-static content, and suggested that designers should move away from interactivity . However, other research found that many readers, even those on mobile devices, are interested in utilizing interactivity when it is a core part of the article’s message . This statement from *The New York Times* has solidified as a rule-of-thumb for designers and many choose not to utilize interactivity because of it, despite follow-up discussion that contextualizes the original point and highlights scenarios where interactivity can be beneficial . This means designers are potentially choosing a suboptimal presentation of their story due to this anecdote. More research is needed in order to identify the cases in which interactivity is worth the cost of creation.

 We believe in the power and untapped potential of interactive articles for sparking reader’s desire to learn and making complex ideas accessible and understandable to all.

Communicating with Interactive Articles

Fred Hohman
Matthew Conlen
Jeffrey Heer
Duen Horng (Polo) Chau

Summary: Examining the design of interactive articles by synthesizing theory from disciplines such as education, journalism, and visualization.

Report on the role of digital access providers

A personal computer for children of all ages

Augmenting human intellect: A conceptual framework

The knowledge navigator

PLATO

PhET interactive simulations

Explorable explanations

How y’all, youse and you guys talk

Snow fall: The avalanche at tunnel creek

Why outbreaks like coronavirus spread exponentially, and how to 'flatten the curve'

Attacking discrimination with smarter machine learning

Coeffects: Context-aware programming languages

Complexity explained

What's really warming the world

You draw it: How family income predicts children’s college chances

The Uber Game

Let's learn about waveforms

The book of shaders

EconGraphs

To build a better ballot

The atlas of redistricting

Is it better to rent or buy

Explorable explanation

Increasing the transparency of research papers with explorable multiverse analyses

Workshop on Visualization for AI Explainability

Exploranation: A new science communication paradigm

Research debt

More than telling a story: Transforming data into visually shared stories

"Concrete" computer manipulatives in mathematics education

Interactive non-fiction: Towards a new approach for storytelling in digital journalism

Active essays on the web

Simply bells and whistles? Cognitive effects of visual aesthetics in digital longforms

Learning as a function of time

Emotional design in multimedia learning

Hooked on data videos: assessing the effect of animation and pictographs on viewer engagement

Animation: Can it facilitate?

Animated transitions in statistical data graphics

Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering

The horse in motion

Emergent tool use from multi-agent autocurricula

Climate spirals

Earth's relentless warming sets a brutal new record in 2017

Global temperature

It's not your imagination. Summers are getting hotter.

Extensive data shows punishing reach of racism for black boys

Disagreements

Gun deaths in america

The fallen of World War II

Showing people behind data: Does anthropomorphizing visualizations elicit more empathy for human rights data?

Beyond memorability: Visualization recognition and recall

What makes a visualization memorable?

What if the data visualization is actually people

Ethical dimensions of visualization research

A walk among the data

Visual narrative flow: Exploring factors shaping data visualization story reading experiences

Linking and layout: Exploring the integration of text and visualization in storytelling

Cutthroat Capitalism: The Game

Combining software games with education: Evaluation of its educational effectiveness

Narrative visualization: Telling stories with data

Parable of the polygons

Drawing dynamic visualizations

Scientific communication as sequential art

How you will die

On Particle Physics

Should prison sentences be based on crimes that haven’t been committed yet

How to use t-SNE effectively

The beginner's guide to dimensionality reduction

Understanding UMAP

Tensorflow.js: Machine learning for the web and beyond

Designing (and learning from) a teachable machine

Experiments in handwriting with a neural network

Gan lab: Understanding complex deep generative models using interactive visual experimentation

Using artificial intelligence to augment human intelligence

Who will win the presidency

Who will be president

Live results: Presidential election

Multimedia learning

3Blue1Brown

Primer

Revising the redundancy principle in multimedia learning

Visualizing quaternions: An explorable video series

You draw it: Just how bad is the drug overdose epidemic

You draw it: What got better or worse during Obama's presidency

They draw it!

Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation

The Gyllenhaal experiment

How do you draw a circle? We analyzed 100,000 drawings to show how culture shapes our instincts

The power of testing memory: Basic research and implications for educational practice

Khan Academy

The critical importance of retrieval for learning

How to remember anything for forever-ish

Quantum country

Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice.

Authoring and generation of individualized patient education materials

PersaLog: Personalization of news article content

Generating personalized spatial analogies for distances and areas

How much hotter is your hometown than when you were born

Human terrain

Are you rich? This income-rank quiz might change how you see yourself

Quiz: Let us predict whether you’re a democrat or a republican

Find Out If Your Job Will Be Automated

Booze calculator: What's your drinking nationality

Click 1,000: How the pick-your-own-path episode was made

Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds?

Pictorial aids for learning by doing in a multimedia geology simulation game.

A visual introduction to machine learning

Beyond guidelines: What can we learn from the visual information seeking mantra?

The eyes have it: A task by data type taxonomy for information visualizations

Information visualization and visual data mining

How the recession shaped the economy, in 255 charts

How does the eye work

Earth primer

Progressive growing of gans for improved quality, stability, and variation

A style-based generator architecture for generative adversarial networks

Why momentum really works

Colorized math equations

Image kernels

On variable level-of-detail documents

Call for proposals winter/spring 2019

A UI that lets readers control how much information they see

Wikipedia Preview Card

Fluid links for informed and incremental link transitions

Reading and writing fluid hypertext narratives

Idyll: A markup language for authoring and publishing interactive articles on the web

Apparatus: A hybrid graphics editor and programming environment for creating interactive diagrams

Observable

LOOPY: a tool for thinking in systems

Webstrates: shareable dynamic media

Neural networks and deep learning

How I make explorable explanations

Explorable explanations: 4 more design patterns

Emerging and recurring data-driven storytelling techniques: Analysis of a curated collection of recent stories

Issue 01: Science & Society

The myth of the impartial machine

D3 data-driven documents

Launching the Parametric Press

Techniques for flexible responsive visualization design

A Comparative Evaluation of Animation and Small Multiples for Trend Visualization on Mobile Phones

Visualizing ranges over time on mobile phones: a task-based crowdsourced evaluation

Why we are doing fewer interactives

Capture & analysis of active reading behaviors for interactive articles on the web

In defense of interactive graphics


 Self-organisation is omnipresent on all scales of biological life. From complex interactions between molecules
 forming structures such as proteins, to cell colonies achieving global goals like exploration by means of the
 individual cells collaborating and communicating, to humans forming collectives in society such as tribes,
 governments or countries. The old adage “the whole is greater than the sum of its parts”, often ascribed to
 Aristotle, rings true everywhere we look.

 The articles in this thread focus on practical ways of designing self-organizing systems. In particular we use
 Differentiable Programming (optimization) to learn agent-level policies that satisfy system-level objectives. The
 cross-disciplinary nature of this thread aims to facilitate ideas exchange between ML and developmental biology
 communities.

Articles & Comments
-------------------

 Distill has invited several researchers to publish a “thread” of short articles exploring differentiable
 self-organizing systems,
 interspersed with critical commentary from several experts in adjacent fields.
 The thread will be a living document, with new articles added over time.
 Articles and comments are presented below in chronological order:

###
[Growing Neural Cellular Automata](/2020/growing-ca/)

### Authors

### Affiliations

[Alexander Mordvintsev](https://znah.net/),
 Ettore Randazzo,
 [Eyvind Niklasson](https://eyvind.me/),
 [Michael Levin](http://www.drmichaellevin.org/)

[Google](https://research.google/),
 [Allen Discovery Center](https://allencenter.tufts.edu/)

 Building their own bodies is the very first skill all living creatures possess. How can we design systems that
 grow, maintain and repair themselves by regenerating damages? This work investigates morphogenesis, the
 process by which living creatures self-assemble their bodies. It proposes a differentiable, Cellular Automata
 model of morphogenesis and shows how such a model learns a robust and persistent set of dynamics to grow any
 arbitrary structure starting from a single cell.
 [Read Full Article](/2020/growing-ca/)

###
[Self-classifying MNIST Digits](/2020/selforg/mnist/)

### Authors

### Affiliations

Ettore Randazzo,
 [Alexander Mordvintsev](https://znah.net/),
 [Eyvind Niklasson](https://eyvind.me/),
 [Michael Levin](http://www.drmichaellevin.org/),
 [Sam Greydanus](https://greydanus.github.io/about.html)

[Google](https://research.google/),
 [Allen Discovery Center](https://allencenter.tufts.edu/),
 [Oregon State University and the ML Collective](http://mlcollective.org/)

 This work presents a follow up to Growing Neural CAs, using a similar computational model for the goal of
 digit “self-classification”. The authors show how neural CAs can self-classify the MNIST digit they form. The
 resulting CAs can be interacted with by dynamically changing the underlying digit. The CAs respond to
 perturbations with a learned self-correcting classification behaviour.
 [Read Full Article](/2020/selforg/mnist/)

###
[Self-Organising Textures](/selforg/2021/textures/)

### Authors

### Affiliations

[Eyvind Niklasson](https://eyvind.me/),
 [Alexander Mordvintsev](https://znah.net/),
 Ettore Randazzo,
 [Michael Levin](http://www.drmichaellevin.org/)

[Google](https://research.google/),
 [Allen Discovery Center](https://allencenter.tufts.edu/),

 Here the authors apply Neural Cellular Automata to a new domain: texture synthesis. They begin by training NCA
 to mimic a series of textures taken from template images. Then, taking inspiration from adversarial
 camouflages which appear in nature, they use NCA to create textures which maximally excite neurons in a
 pretrained vision model. These results reveal that a simple model combined with well-known objectives can lead
 to robust and unexpected behaviors.
 [Read Full Article](/selforg/2021/textures/)

###
[Adversarial Reprogramming of Neural Cellular Automata](/selforg/2021/adversarial/)

### Authors

### Affiliations

[Ettore Randazzo](https://oteret.github.io/),
 [Alexander Mordvintsev](https://znah.net/),
 [Eyvind Niklasson](https://eyvind.me/),
 [Michael Levin](http://www.drmichaellevin.org/)

[Google](https://research.google/),
 [Allen Discovery Center](https://allencenter.tufts.edu/),

This work takes existing Neural CA models and shows how they can be adversarially reprogrammed to perform novel tasks.
MNIST CA can be deceived into outputting incorrect classifications and the patterns in Growing CA can be made to have their shape and colour altered.
 [Read Full Article](/selforg/2021/adversarial/)

#### This is a living document

 Expect more articles on this topic, along with critical comments from
 experts.

Get Involved
------------

 The Self-Organizing systems thread is open to articles exploring differentiable self-organizing sytems.
 Critical
 commentary and discussion of existing articles is also welcome. The thread
 is organized through the open `#selforg` channel on the
 [Distill slack](http://slack.distill.pub). Articles can be
 suggested there, and will be included at the discretion of previous
 authors in the thread, or in the case of disagreement by an uninvolved
 editor.

 If you would like get involved but don’t know where to start, small
 projects may be available if you ask in the channel.

About the Thread Format
-----------------------

 Part of Distill’s mandate is to experiment with new forms of scientific
 publishing. We believe that that reconciling faster and more continuous
 approaches to publication with review and discussion is an important open
 problem in scientific publishing.

 Threads are collections of short articles, experiments, and critical
 commentary around a narrow or unusual research topic, along with a slack
 channel for real time discussion and collaboration. They are intended to
 be earlier stage than a full Distill paper, and allow for more fluid
 publishing, feedback and discussion. We also hope they’ll allow for wider
 participation. Think of a cross between a Twitter thread, an academic
 workshop, and a book of collected essays.

 Threads are very much an experiment. We think it’s possible they’re a
 great format, and also possible they’re terrible. We plan to trial two
 such threads and then re-evaluate our thought on the format.

Thread: Differentiable Self-organizing Systems

Alexander Mordvintsev
Ettore Randazzo
Eyvind Niklasson
Michael Levin
Sam Greydanus
Alexander Mordvintsev
Ettore Randazzo
Eyvind Niklasson
Michael Levin
Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin
Sam Greydanus
Eyvind Niklasson
Alexander Mordvintsev
Ettore Randazzo
Michael Levin
Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin

Summary: A collection of articles and comments with the goal of understanding how to design robust and general purpose self-organizing systems.


### Contents

[Model](#model)
[Experiments](#experiment-1)
* [Self-classify, persist & mutate](#experiment-1)
* [Stabilizing classification](#experiment-2)

[Robustness](#robustness)
[Related Work](#related-work)
[Discussion](#discussion)

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Growing Neural Cellular Automata](/2020/growing-ca/)
[Self-Organising Textures](/selforg/2021/textures/)

Growing Neural Cellular Automata  demonstrated how simple cellular automata (CAs) can learn to self-organise into complex shapes while being resistant to perturbations. Such a computational model approximates a solution to an open question in biology, namely, how do cells cooperate to create a complex multicellular anatomy and work to regenerate it upon damage? The model parameterizing the cells’ rules is parameter-efficient, end-to-end differentiable, and illustrates a new approach to modeling the regulation of anatomical homeostasis. In this work, we use a version of this model to show how CAs can be applied to a common task in machine learning: classification. We pose the question: *can CAs use local message passing to achieve global agreement* *on* *what digit they compose?*

Our question is closely related to another unsolved problem in developmental and regenerative biology: how cell groups decide whether an organ or tissue pattern is correct, or whether current anatomy needs to be remodeled (anatomical surveillance and repair toward a specific target morphology). For example, when scientists surgically transplanted a salamander tail to its flank, it slowly remodeled into a limb - the organ that belongs at this location . Similarly, tadpoles with craniofacial organs in the wrong positions usually become normal frogs because they remodel their faces, placing the eye, mouth, nostrils, etc. in their correct locations. Cell groups move around and stop when the correct frog-specific anatomical configuration has been achieved . All of these examples illustrate the ability of biological systems to determine their current anatomical structure and decide whether it matches a species-specific target morphology . Despite the recent progress in molecular biology of genes necessary for this process, there is still a fundamental knowledge gap concerning the algorithms sufficient for cell collectives to measure and classify their own large-scale morphology. More broadly, it is important to create computational models of swarm intelligence that explicitly define and distinguish the dynamics of the basal cognition of single cells versus cell collectives .

### The self-classifying MNIST task

Suppose a population of agents is arranged on a grid. They do not know where they are in the grid and they can only communicate with their immediate neighbors. They can also observe whether a neighbor is missing. Now suppose these agents are arranged to form the shape of a digit. Given that all the agents operate under the same rules, can they form a communication protocol such that, after a number of iterations of communication, *all of the agents know which digit they are forming**?*Furthermore, if some agents were to be removed and added to form a new digit from a preexisting one, would they be able to know which the new digit is?

Because digits are not rotationally invariant (i.e. 6 is a rotation of 9), we presume the agents must be made aware of their orientation with respect to the grid. Therefore, while they do not know where they are, they do know where up, down, left and right are. The biological analogy here is a situation where the remodeling structures exist in the context of a larger body and a set of morphogen gradients or tissue polarity that indicate directional information with respect to the three major body axes. Given these preliminaries, we introduce the self-classifying MNIST task.

A visualisation of a random sample of digits from MNIST, each shaded by the colour corresponding its label.

Each sample of the MNIST dataset  consists of a 28x28 image with a single monochrome channel that is classically displayed in greyscale. The label is an integer in [0,9][0,9][0,9].

Our goal is for all cells that make up the digit to correctly output the label of the digit. To convey this structural information to the cells, we make a distinction between alive and dead cells by rescaling the values of the image to [0, 1]. Then we treat a cell as alive if its value in the MNIST sample is larger than 0.1. The intuition here is that we are placing living cells in a cookie cutter and asking them to identify the global shape of the cookie cutter. We visualize the label output by assigning a color to each cell, as you can see above. We use the same mapping between colors and labels throughout the article. Please note that there is a slider in the interactive demo controls which you can use to adjust the color palette if you have trouble differentiating between the default colors.

Model
-----

In this article, we use a variant of the neural cellular automata model described in Growing Cellular Automata . We refer readers unfamiliar with its implementation to the original [”Model”](https://distill.pub/2020/growing-ca/#model) section. Here we will describe a few areas where our model diverges from the original.

### Target labels

The work in Growing CA used RGB images as targets, and optimized the first three state channels to approximate those images. For our experiments, we treat the last ten channels of our cells as a pseudo-distribution over each possible label (digit). During inference, we simply pick the label corresponding to the channel with the highest output value.

### Alive cells and cell states

In Growing CA we assigned a cell’s state to be “dead” or “alive” based on the strength of its alpha channel and the activity of its neighbors. This is similar to the rules of Conway’s Game of Life . In the Growing CA model, “alive” cells are cells which update their state and dead cells are “frozen” and do not undergo updates. In contrast to biological life, what we call “dead” cells aren’t dead in the sense of being non-existent or decayed, but rather frozen: they are visible to their neighbors and maintain their state throughout the simulation. In this work, meanwhile, we use input pixel values to determine whether cells are alive or dead and perform computations with alive cells only As introduced in the previous section, cells are considered alive if their normalized grey value is larger than 0.1.. It is important to note that the values of MNIST pixels are exposed to the cell update rule as an immutable channel of the cell state. In other words, we make cells aware of their own pixel intensities as well as those of their neighbors. Given 19 mutable cell state channels (nine general purpose state channels for communication and ten output state channels for digit classification) and an immutable pixel channel, each cell perceives 19 + 1 state channels and only outputs state updates for the 19 mutable state channels.

**A note on digit topology.**Keen readers may notice that our model requires each digit to be a single connected component in order for classification to be possible, since any disconnected components will be unable to propagate information between themselves. We made this design decision in order to stay true to our core biological analogy, which involves a group of cells that is trying to identify its global shape. Even though the vast majority of samples from MNIST are fully connected, some aren’t. We do not expect our models to classify non-connected minor components correctly, but we do not remove them This choice complicates comparison between the MNIST train/test accuracies of neural network classifiers vs. CAs. However, such a comparison is not in scope of this article..

### Perception

The Growing CA article made use of fixed 3x3 convolutions with Sobel filters to estimate the state gradients in x⃗\vec{x}x⃗ and y⃗\vec{y}y⃗. We found that fully trainable 3x3 kernels outperformed their fixed counterparts and so used them in this work.

**A note on model size.** Like the Growing CA model, our MNIST CA is small by the standards of deep learning - it has less than 25k parameters. Since this work aims to demonstrate a novel approach to classification, we do not attempt to maximise the validation accuracy of the model by increasing the number of parameters or any other tuning. We suspect that, as with other deep neural network models, one would observe a positive correlation between accuracy and model size.

Experiment 1: Self-classify, persist and mutate
-----------------------------------------------

In our first experiment, we use the same training paradigm as was discussed in Growing CA. We train with a pool of initial samples to allow the model to learn to persist and then perturb the converged states. However, our perturbation is different. Previously, we destroyed the states of cells at random in order to make the CAs resistant to destructive perturbations (analogous to traumatic tissue loss). In this context, perturbation has a slightly different role to play. Here we aim to build a CA model that not only has regenerative properties, but also *has the ability to correct itself when the shape of the overall digit changes**.*

Biologically, this corresponds to a teratogenic influence during development, or alternatively, a case of an incorrect or incomplete remodeling event such as metamorphosis or rescaling. The distinction between training our model from scratch and training it to accommodate perturbations is subtle but important. An important feature of life is the ability to react adaptively to external perturbations that are not accounted for in the normal developmental sequence of events. If our virtual cells simply learned to recognize a digit and then entered some dormant state and did not react to any further changes, we would be missing this key property of living organisms. One could imagine a trivial solution in the absence of perturbations, where a single wave of information is passed from the boundaries of the digit inwards and then back out, in such a way that all cells could agree on a correct classification. By introducing perturbations to new digits, the cells have to be in constant communication and achieve a “dynamic homeostasis” - continually “kept on their toes” in anticipation of new or further communication from their neighbours.

In our model, we achieve this dynamic homeostasis by randomly mutating the underlying digit at training time. Starting from a certain digit and after some time evolution, we sample a new digit, erase all cell states that are not present in both digits and bring alive the cells that were not present in the original digit but are present in the new digit. This kind of mutation teaches CAs to learn to process new information and adapt to changing conditions. It also exposes the cells to training states where all of the cells that remain after a perturbation are misclassifying the new digit and must recover from this catastrophic mutation. This in turn forces our CAs to learn to change their own classifications to adapt to changing global structure.

We use a pixel-wise (cell-wise) cross entropy loss on the last ten channels of each pixel, applying it after letting the CA evolve for 20 steps.

 Your browser does not support the video tag.

A first attempt at having the neural CAs classify digits. Each digit is a separate evolution of the neural CA, with the visualisations collated. Halfway through, the underlying digit is swapped for a new one - a “mutation”.

The video above shows the CA classifying a batch of digits for 200 steps. We then mutate the digits and let the system evolve and classify for a further 200 steps.

The results look promising overall and we can see how our CAs are able to recover from mutations. However, astute observers may notice that often not all cells agree with each other. Often, the majority of the digit is classified correctly, but some outlier cells are still convinced they are part of a different digit, often switching back and forth in an oscillating pattern, causing a flickering effect in the visualization. This is not ideal, since we would like the population of cells to reach stable, total, agreement. The next experiment troubleshoots this undesired behaviour.

Experiment 2: Stabilizing classification
----------------------------------------

Quantifying a qualitative issue is the first step to solving it. We propose a metric to track **average cell** **accuracy**, which we define as the mean percentage of cells that have a correct output. We track this metric both before and after mutation.

Average accuracy across the cells in a digit over time.

In the figure above, we show the mean percentage of correctly classified pixels in the test set over the course of 400 steps. At step 200, we randomly mutate the digit. Accordingly, we see a brief drop in accuracy as the cells re-organise and eventually come to agreement on what the new digit is.

We immediately notice an interesting phenomenon: the cell accuracy appears to decrease over time after the cells have come to an agreement. However, the graph does not necessarily reflect the qualitative issue of unstable labels that we set out to solve. The slow decay in accuracy may be a reflection of the lack of total agreement, but doesn’t capture the stark instability issue.

Instead of looking at the mean agreement perhaps we should measure **total agreement**. We define total agreement as the percentage of samples from a given batch wherein all the cells output the same label.

Average total agreement among cells across the test set in MNIST, over time.

This metric does a better job of capturing the issues we are seeing. The total agreement starts at zero and then spikes up to roughly 78%, only to lose more than 10% agreement over the next 100 steps. Again, behaviour after mutation does not appear to be significantly different. Our model is not only unstable in the short term, exhibiting flickering, but is also unstable over longer timescales. As time goes on, cells are becoming less sure of themselves. Let’s inspect the inner states of the CA to see why this is happening.

Average magnitude of the state channels and residual updates in active cells over time in the test set.

The figure above shows the time evolution of the average magnitude of the state values of active cells (solid line), and the average magnitude of the residual updates (dotted line). Two important things are happening here: 1) the average magnitude of each cell’s internal states is growing monotonically on this timescale; 2) the average magnitude of the residual updates is staying roughly constant. We theorize that, unlike 1), a successful CA model should stabilize the magnitude of its internal states once cells have reached an agreement. In order for this to happen, its residual updates should approach zero over time, unlike what we observed in 2).

**Using an** **L2L\_2L2** **loss.** One problem with cross entropy loss is that it tends to push raw logit values indefinitely higher. Another problem is that two sets of logits can have vastly different values but essentially the same prediction over classes. As such, training the CA with cross-entropy loss neither requires nor encourages a shared reference range for logit values, making it difficult for the cells to effectively communicate and stabilize. Finally, we theorize that large magnitudes in the classification channels may in turn lead the remaining (non-classification) state channels to transition to a high magnitude regime. More specifically, we believe that *cross-entropy loss causes unbounded growth in classification logits, which prevents residual updates from approaching zero, which means that neighboring cells continue passing messages to each other even after they reach an agreeme**nt**.**Ultimately, this causes the magnitude of the message vectors to grow unboundedly*. With these problems in mind, we instead try training our model with a pixel-wise L2L\_2L2 loss and use one-hot vectors as targets. Intuitively, this solution should be more stable since the raw state channels for classification are never pushed out of the range [0,1][0, 1][0,1] and a properly classified digit in a cell will have exactly one classification channel set to 1 and the rest to 0. In summary, an L2L\_2L2 loss should decrease the magnitude of all the internal state channels while keeping the classification targets in a reasonable range.

**Adding noise to the residual updates**. A number of popular regularization schemes involve injecting noise into a model in order to make it more robust . Here we add noise to the residual updates by sampling from a normal distribution with a mean of zero and a standard deviation of 2×10−22 \times 10^{-2}2×10−2. We add this noise before randomly masking the updates.

 Your browser does not support the video tag.

Neural CA trained with L2L\_2L2 loss, exhibiting less instability after converging to a label.

The video above shows a batch of runs with the augmentations in place. Qualitatively, the result looks much better as there is less flickering and more total agreement. Let’s check the quantitative metrics to see if they, too, show improvement.

Comparison of average accuracy and total agreement
 when using cross-entropy and when using L2L\_2L2 loss.

| Model | Top accuracy | Accuracy at 200  | Top agreement | Agreement at 200 |
| --- | --- | --- | --- | --- |
| CE | **96.2 at 80** | **95.3** | 77.9 at 80 | 66.2 |
| L2L\_2L2 | 95.0 at 95 | 94.7 | 85.5 at 175 | 85.2 |
| L2L\_2L2 + Noise | 95.4 at 65 | **95.3** | **88.2 at 190** | **88.1** |

The figure and table above show that cross-entropy achieves the highest accuracy of all models at roughly 80 steps. However, the accuracy at 200 steps is the same as the L2L\_2L2 + Noise model. While accuracy and agreement degrade over time for all models, the L2L\_2L2 + Noise appears to be the most stable configuration. In particular, note that the total agreement after 200 steps of L2L\_2L2 + Noise is 88%, an improvement of more than 20% compared to the cross-entropy model.

### Internal states

Average magnitude of state channels over time for L2L\_2L2 loss and cross-entropy loss.

Let’s compare the internal states of the augmented model to those of the original. The figure above shows how switching to an L2L\_2L2 loss stabilizes the magnitude of the states, and how residual updates quickly decay to small values as the system nears agreement.

 Your browser does not support the video tag.

Visualisation of internal state channel values during mutations. Note the accelerated timeline after a few seconds showing the relative stability of the channel values.

To further validate our results, we can visualize the dynamics of the internal states of the final model. For visualization purposes, we have squashed the internal state values by applying an element-wise arctanarctanarctan, as most state values are less than one but a few are much larger. The states converge to stable configurations quickly and the state channels exhibit spatial continuity with the neighbouring states. More specifically, we don’t see any stark discontinuities in state values of neighbouring pixels. Applying a mutation causes the CA to readapt to the new shape and form a new classification in just a few steps, after which its internal values are stable.

Robustness
----------

Recall that during training we used random digit mutations to ensure that the resulting CA would be responsive to external changes. This allowed us to learn a dynamical system of agents which interact to produce stable behavior at the population level, even when perturbed to form a different digit from the original. Biologically, this model helps us understand the mutation insensitivity of some large-scale anatomical control mechanisms. For example, planaria continuously accumulate mutations over millions of years of somatic inheritance but still always regenerate the correct morphology in nature (and exhibit no genetic strains with new morphologies) .

This robustness to change was critically important to our interactive demo, since the cells needed to reclassify drawings as the user changed them. For example, when the user converted a six to an eight, the cells needed to quickly re-classify themselves to an eight. We encourage the reader to play with the interactive demo and experience this for themselves. In this section, we want to showcase a few behaviours we found interesting.

 Your browser does not support the video tag.

Demonstration of the CA successfully re-classifying a digit when it is modified by hand.

The video above shows how the CA is able to interactively adjust to our own writing and to change classification when the drawing is updated.

### Robustness to out-of-distribution shapes

In the field of machine learning, researchers take great interest in how their models perform on out-of-distribution data. In the experimental sections of this article, we evaluated our model on the test set of MNIST. In this section, we go further and examine how the model reacts to digits drawn by us and not sampled from MNIST at all. We vary the shapes of the digits until the model is no longer capable of classifying them correctly. Every classification model inherently contains certain inductive biases that render them more or less robust to generalizing to out-of-distribution data. Our model can be seen as a recurrent convolutional model and thus we expect it to exhibit some of the key properties of traditional convolutional models such as translation invariance. However, we strongly believe that the self-organising nature of this model introduces a novel inductive bias which may have interesting properties of its own. Biology offers examples of “repairing to novel configurations”: 2-headed planaria, once created, regenerate to this new configuration which was not present in the evolutionary “training set” .

 Your browser does not support the video tag.

Demonstration of some of the failure cases of the CA.

Above, we can see that our CA fails to classify some variants of 1 and 9. This is likely because MNIST training data is not sufficiently representative of all writing styles. We hypothesize that more varied and extensive datasets would improve performance. The model often oscillates between two attractors (of competing digit labels) in these situations. This is interesting because this behavior could not arise from static classifiers such as traditional convolutional neural networks.

 Your browser does not support the video tag.

Demonstration of the inherent robustness of the model to unseen sizes and variants of numbers.

By construction, our CA is translation invariant. But perhaps surprisingly, we noticed that our model is also scale-invariant for out-of-distribution digit sizes up to a certain point. Alas, it does not generalize well enough to classify digits of arbitrary lengths and widths.

 Your browser does not support the video tag.

Demonstration of the behaviour of the model with chimeric configurations.

It is also interesting to see how our CA classifies “chimeric digits”, which are shapes composed of multiple digits. First, when creating a 3-5 chimera, the classification of 3 appears to dominate that of the 5. Second, when creating a 8-9 chimera, the CAs reach an oscillating attractor where sections of the two digits are correctly classified. Third, when creating a 6-9 chimera, the CAs converge to an oscillating attractor but the 6 is misclassified as a 4.

These phenomena are important in biology as scientists begin to develop predictive models for the morphogenetic outcome of chimeric cell collectives. We still do not have a framework for knowing in advance what anatomical structures will form from a combination of, for example leg-and-tail blastema cells in an axolotl, heads of planaria housing stem cells from species with different head shapes, or composite embryos consisting of, for example, frog and axolotl blastomeres . Likewise, designing information signals that induce the emergence of desired tissue patterns from a chimeric cellular collective, in vitro or in vivo, remains an open problem.

Related Work
------------

This article is follow-up work to Growing Neural Cellular Automata , and it is meant to be read after the latter. In this article, we purposefully skim over details of the original model and we refer the reader to the Growing Neural Cellular Automata article for the [full model description](https://distill.pub/2020/growing-ca/#model) section and [related work](https://distill.pub/2020/growing-ca/#related-work) section.

**MNIST and CA.** Since CAs are easy to apply to two dimensional grids, many researchers wondered if they could use them to somehow classify the MNIST dataset. We are aware of work that combines CAs with Reservoir Computing , Boltzmann Machines , Evolutionary Strategies , and ensemble methods . To the best of our knowledge, we are the first to train end-to-end differentiable Neural CAs for classification purposes and we are the first to introduce the self-classifying variant of MNIST wherein each pixel in the digit needs to coordinate locally in order to reach a global agreement about its label.

Discussion
----------

This article serves as a proof-of-concept for how simple self-organising systems such as CA can be used for classification when trained end-to-end through backpropagation.

Our model adapts to writing and erasing and is surprisingly robust to certain ranges of digit stretching and brush widths. We hypothesize that self-organising models with constrained capacity may be inherently robust and have good generalisation properties. We encourage future work to test this hypothesis.

From a biological perspective, our work shows we can teach things to a collective of cells that they could not learn individually (by training or engineering a single cell). Training cells in unison (while communicating with each other) allows them to learn more complex behaviour than any attempt to train them one by one, which has important implications for strategies in regenerative medicine. The current focus on editing individual cells at the genetic or molecular signaling level faces fundamental barriers when trying to induce desired complex, system-level outcomes (such as regenerating or remodeling whole organs). The inverse problem of determining which cell-level rules (e.g., genetic information) must be changed to achieve a global outcome is very difficult. In contrast and complement to this approach, we show the first component of a roadmap toward developing effective strategies for communication with cellular collectives. Future advances in this field may be able to induce desired outcomes by using stimuli at the system’s input layer (experience), not hardware rewiring, to re-specify outcomes at the tissue, organ, or whole-body level .

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Growing Neural Cellular Automata](/2020/growing-ca/)
[Self-Organising Textures](/selforg/2021/textures/)

Self-classifying MNIST Digits

Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin
Sam Greydanus

Summary: Training an end-to-end differentiable, self-organising cellular automata for classifying MNIST digits.

Growing Neural Cellular Automata

The transformation of a tail into limb after xenoplastic transplantation

Normalized shape and location of perturbed craniofacial structures in the Xenopus tadpole reveal an innate ability to achieve correct morphology

Top-down models in biology: explanation and control of complex living systems above the molecular level

MATHEMATICAL GAMES

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Practical Variational Inference for Neural Networks

Planarian regeneration as a model of anatomical homeostasis: Recent progress in biophysical and computational approaches

Long-range neural and gap junction protein-mediated cues control polarity during planarian regeneration

Bioelectrical Mechanisms for Programming Growth and Form: Taming Physiological Networks for Soft Body Robotics

Interspecies chimeras

Asynchronous network of cellular automaton-based neurons for efficient implementation of Boltzmann machines

Biologically inspired cellular automata learning and prediction model for handwritten pattern recognition


Curve Detectors

Nick Cammarata
Gabriel Goh
Shan Carter
Ludwig Schubert
Michael Petrov
Chris Olah

Summary: Part one of a three part deep dive into the curve neuron family.

Visualizing and understanding convolutional networks

Distributed representations of words and phrases and their compositionality

Visualizing and understanding recurrent networks

Learning to generate reviews and discovering sentiment

Object detectors emerge in deep scene cnns

Feature Visualization

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Measuring the tendency of CNNs to Learn Surface Statistical Regularities

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Approximating cnns with bag-of-local-features models works surprisingly well on imagenet

Adversarial examples are not bugs, they are features

On the importance of single directions for generalization

Visualizing higher-layer features of a deep network

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Axiomatic attribution for deep networks

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (Un)reliability of saliency methods

Differentiable Image Parameterizations

Discrete neural clusters encode orientation, curvature and corners in macaque V4

Synthetic Abstractions

The Building Blocks of Interpretability

Using Artificial Intelligence to Augment Human Intelligence