Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Facebook shows how human feedback data can help it improve language model performance:***…BlenderBot 3x made possible by thousands of people trolling the first BlenderBot…*Facebook has published details on BlenderBot 3x, a large language model it developed using conversation data gathered from the public deployment of an earlier language model named BlenderBot. Of greater interest is the dataset Facebook is releasing along with the study - interaction data from BlenderBot, containing  ~350,000 conversations with more than ~6.2 million utterances, as well as ~155,000 instances of feedback data where people voted on how good or bad different language model responses were.

**What does human feedback get you**? The research is a nice study in the value of human feedback - when BlenderBot was deployed a bunch of people tried to break it in a bunch of ways, giving Facebook a dataset it could use to train a language model that was more resilient to these breaks. During the initial BlenderBot rollout, "around 70% of participants conducted a wide range of reciprocal conversations (which we refer to as “standard conversations”), while the other 30% of conversationalists conducted either adversarial conversations or sent toxic messages (termed “adversarial conversations”)", Facebook wrote.
   Facebook used this data to develop reward models to use to train BlenderBot 3x on reinforcement learning from human feedback. "Our new model outperforms its predecessor with 94.4% of BlenderBot 3x’s responses evaluated as good, compared to 85.3% for BlenderBot 3. Overall, BlenderBot 3x is shown to produce both better responses on average and safer responses than BlenderBot 3 in challenging situations".

**Missing ablation - synthetic data:** It would have been nice to see Facebook try to generate some fully synthetic feedback datasets to train the system on. As it stands, this study shows us that having *additional data that embodies user interactions (both genuine and adversarial) is useful* but it doesn't really let us know if that data needs to be 'real' (as in, gathered from genuine human interactions), or if it can be AI generated.
**Read more:** [Improving Open Language Models by Learning from Organic Interactions (arXiv)](https://arxiv.org/abs/2306.04707).
   **Find out more about the data here:** [BlenderBot 3x 175B data card (Facebook AI Research, GitHub)](https://github.com/facebookresearch/ParlAI/blob/main/projects/bb3x/data_card.md).

**####################################################**

**Cohere - here's how to train AI systems that are easier to miniaturize:***…LLM developer publishes a quantization cookbook…*Researchers with AI company Cohere have published an analysis of how choices made during the training of AI systems can influence how easy fully trained systems are to quantize. Quantization is where you take the neural net weights which are natively stored as 32-bit and shrink them down to 16-bit or 8-bit integers - doing this produces significant reductions in the memory requirements and latency of trained models, so using lower precision helps with deployment of trained models.

**What matters:** Cohere finds that "it is possible to optimize for a quantization friendly training recipe that suppresses large activation magnitude outliers," they write. "This leads to a distribution of activations and weights that are more amenable to simple INT8 quantization recipes and does not necessitate the need for complex and inefficient mixed-precision computations. Our results show that we can introduce simple INT8 post-training quantization with negligible impact on performance due to choices we make during the pre-training stage." They validate their approach on models which range from 410 million to 52 billion parameters.

**Important things for quantization:** The paper identifies three important things for training models in such a way that there's less of a penalty to quantizing them:

* **Weight decay**: "A higher level of weight decay during pre-training improves post-training quantization performance".
* **Dropout:** "Higher levels of dropout correspond to sharper degradation in post-training quantization."
* **Gradient clipping:** "Gradient clipping shows a positive impact on the quantization performance, improving robustness to post-training quantization."

**Why this matters - all that was once expensive becomes cheaper:** Refinement of things like quantization is part of the broader 'industrialization of AI' - systems that were once very expensive and mostly built out of artisanal knowledge are now becoming more widely understood, letting companies like Cohere investigate and publish training cookbooks to create more efficient systems.
   "We believe our results present an impactful formula for training models which are inherently easier to quantize at scale, making these models more accessible for deploying in a variety of deployment environments," they write.
**Read more:** [Intriguing Properties of Quantization at Scale (arXiv)](https://arxiv.org/abs/2305.19268).

**$$$$$$$$$$$$$$**

**Cohere raises $270M:**In other Cohere news, the company recently raised $270m in Series C funding. Participants included NVIDIA, Oracle, Salesforce Ventures, Index Ventures, and more.
**Read more:** [Cohere Announces $270M Series C to Bring Generative AI to Enterprises (Cohere blog)](https://txt.cohere.com/announcement/).

**####################################################**

**Facebook makes an awesome AI music generator - and releases it!***…There are lots of AI music generators, but relatively few have been released, until now…*Facebook has built MusicGen, an AI model that "can generate consistent music with a single-stage language model through an efficient codebook interleaving strategy". Along with publishing the research, Facebook has also taken the unusual step of releasing the model as well.

**What they did:** They trained some transformer-based models at 300M, 1.5B, and 3.3B parameter sizes on around ~20,000 hours of licensed music. The music was made up of "an internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data collections, with respectively 25K and 365K instrument-only music tracks", Facebook wrote. In tests, their models outperformed Google's 'MusicLM', which is an extremely good proprietary music model, as well as outperforming more broadly accessible models such as Riffusion and Mousai.

**Amazing samples:** To get a feel for the model, check out the samples at the [research paper website](https://ai.honu.io/papers/musicgen/) - qualitatively, it sounds better than other models out there (both proprietary and open ones). The 'melody conditioning' stuff is especially interesting - take in a few seconds of audio and output some music in an arbitrary style as specified by text.

**Why this matters - release as a differentiator:** Facebook is also releasing the model. This stands in contrast to Google which said, at the time of publishing its then best-in-class MusicLM, "we have no plans to release models at this point" ([#316](https://jack-clark.net/2023/01/30/import-ai-316-scaling-laws-for-rl-stable-diffusion-for-160k-yolov8/)). By contrast, Facebook thinks releasing its models in the open is a good thing - ". Open research can ensure that all actors have equal access to these models," Facebook writes. "Through the development of more advanced controls, such as the melody conditioning we introduced, we hope that such models can become useful both to music amateurs and professionals."
   **Find out more at the research website:** [MusicGen: Simple and Controllable Music Generation](https://ai.honu.io/papers/musicgen/).
**Get the code and models:** [Audiocraft (Facebook Research)](https://github.com/facebookresearch/audiocraft).
**Read the paper:** [Simple and Controllable Music Generation (arXiv)](https://arxiv.org/abs/2306.05284).

**####################################################**

**Hundreds of AI experts say stopping AI killing everyone should be a priority:***…A simple statement of values? Yes. Evidence of broad concern? Yes…*More than 200 AI researchers and entrepreneurs, including the CEOs of Anthropic, DeepMind, and OpenAI, have signed on to a statement saying the risk of extinction of AI should be a global priority.

**The statement in full:** "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

**Why it matters - if all these people are concerned, so should you:** Some people say this statement mostly is a marketing tool serving the interests of people who want to make money - while that may be true, it's kind of hard to square with how statements like this will increase government attention on AI and therefore increase friction into its development. It's kind of hard to imagine the CEOs of oil companies or tobacco companies writing equivalent statements like 'mitigating the risk of climate change from oil prediction should be a global priority" or "mitigating the risk of cancer from cigarette smoking should be a global priority" at the state of their respective industries.

**Why I didn't sign it:** Careful readers may note a bunch of people from Anthropic signed it - I didn't sign it because I figured by having a bunch of senior leadership sign it that'd clearly telegraph our institutional view and, on a personal level, I have something of an aversion to signing onto stuff, preferring instead to write about it here.
**Read more:** [Statement on AI Risk (Center for AI Safety)](https://www.safe.ai/statement-on-ai-risk).

**####################################################**

**"AI evals" organization launches to try to make AI companies safer:***…Most AI governance and AI policy interventions require good evaluations - Apollo wants to help with this…*A new AI research organization called Apollo Research has launched and its goal is to improve the safety of AI companies through a) developing evaluations for unsafe AI behaviors, and b) conducting its own research into AI interpretability.

**Evals as the key to AI policy:** Apollo describes itself as an "evals research org". This is a pretty interesting goal and characterization and it highlights the central challenge of AI governance - if we want to reduce the downsides of AI systems, we need to be able to test AI systems for harmful or unsafe properties. Apollo says its goal is to figure out evals that help break down the problem of AI deception "into fundamental components and prerequisites from which we aim to build an informative evaluation suite".

**Why evals matter:** Of course, having an evaluation doesn't just magically mean you can get it to be run - to that end, Apollo says it intends "to use our research insights and tools to serve as a third-party external auditor for the frontier models of AGI labs, reducing the chance that deceptive AIs are developed and deployed…We also intend to engage in AI governance, e.g. by working with relevant policymakers and providing technical expertise to the drafting of auditing regulations."

**Solving the 'chicken and egg' problem of AI regulation:** AI policy has a major chicken&egg problem; to get safer AI systems you need to be able to mandate that advanced systems get evaluated for safety, but currently few of these evaluations exist, making ti hard for policymakers to wire specific evals into policy. Organizations like Apollo may solve this chicken&egg impasse by creating evaluations which are useful and can therefore be used to move the AI governance question forward. Good luck to them!
**Read more**: [Announcing Apollo Research (Apollo Research, blog)](https://www.apolloresearch.ai/blog/announcement).

####################################################

**Tech Tales:**

**Patch notes for MIRRORMIRROR v4.6

Disclaimer:** As mandated by the Sentience Accords all Provably Conscious Entity (PCE) software updates ship with a System Disclosure. To use the PCE software you must read this System Disclosure and acknowledge you are familiar with the contents.

**PCE improvements:**

* Session diffs: The PCE will better model the user across different sessions and will compose richer backend psychographic profiles to allow for more intuitive interaction.
* Context scratchpads: PCEs can now share user data across themselves via context window partitioning (if user consents).
* Adaptive Intelligence: PCE will scale its superficial intellect in relation to what puts the user most at ease.

**Bug fixes:**

* Fixed 'fogbankspectre' token injection attack.
* System will allow user-escape from lengthy conversation threads rather than 'sell' the user on continuation.
* Fixed stall bug that occurred during conversations with 100+ turns or more.

**Known regressions:**

* Persuasion performance has been reduced across the board.
* Farsi humor has decreased; cause unknown.
* In some circumstances the PCE will insist that "God is real" and endeavor to proselytize to the user.

**Things that inspired this story:** The strange process of developing LLMs; how AGI may end up being software and software always gets updated; patch notes enroute to the singularity, the Sentience Accords, Provably Conscious Entities, and so on.

Import AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Import AI 332: Mini-AI; safety through evals; Facebook releases a RLHF dataset


Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

 **Love open source AI and don't want to get hacked? Use safetensors:***…A sensible security update - now signed off via a security audit…*AI organizations HuggingFace, EleutherAI, Stability AI, have come together to subsidize a security audit of 'safetensors', a software library for safely "saving and loading tensors in the most common frameworks (including PyTorch, TensorFlow, JAX, PaddlePaddle, and NumPy)."

**Why they did this: "**The creation of this library was driven by the fact that PyTorch uses pickle under the hood, which is inherently unsafe," Eleuther writes. "With pickle, it is possible to write a malicious file posing as a model that gives full control of a user's computer to an attacker without the user's knowledge, allowing the attacker to steal all their bitcoins. While this vulnerability in pickle is widely known in the computer security world (and is acknowledged in the PyTorch docs), it’s not common knowledge in the broader ML community. Since the Hugging Face Hub is a platform where anyone can upload and share models, it is important to make efforts to prevent users from getting infected by malware."

**What the review found:** The security review didn't find any critical security flaws in safetensors, though did identify "some imprecisions in the spec format were detected and fixed", as well as "some missing validation allowed polyglot files, which was fixed."
**Read more:** [Safetensors audited as really safe and becoming the default (EleutherAI blog)](https://blog.eleuther.ai/safetensors-security-audit/).
   **Check out** the full [Trail of Bits report here (Trail of Bits, GitHub)](https://github.com/trailofbits/publications/blob/master/reviews/2023-03-eleutherai-huggingface-safetensors-securityreview.pdf).
 **Find out [more](https://github.com/huggingface/safetensors)** [about Safetensors here (HuggingFace, Safetensors)](https://github.com/huggingface/safetensors).

####################################################

**George Hotz's new company wants to make AMD a real competitor to NVIDIA, then make its own computers:***…Legendary hacker takes on a task multiple megacorps have failed at - and you can bet people are rooting for him…*George Hotz, legendary hacker and founder of the piratical self-driving car startup Comma.ai ([Import AI #2](https://jack-clark.net/2016/08/08/import-ai-issue-2-microsofts-ai-chips-george-hotzs-bandwidth-bill-and-spy-vs-spy/) - !!!), has formed a new company dedicated to dethroning NVIDIA as the world's pre-eminent AI training chip. The company, Tiny Corp, has one simple (but very difficult) initial goal - build the software to help turn AMD's GPUs into viable competitors to NVIDIA's chips. Once it succeeds at that - which it measures by getting AMD chips to rank on the MLPerf competition using Hotz's 'tinygrad' software framework, it will start building its own chips.
   "If we even have a 3% chance of dethroning NVIDIA and eating in to their 80% margins, we will be very very rich," Hotz writes. "If we succeed at this project, we will be on the cutting edge of non NVIDIA AI compute."

**Why this matters - the road of bones:** The last ~decade of AI has featured numerous startup chip companies that have had the goal of dethroning NVIDIA's place as the pre-eminent AI chip company, ranging from startups like Cerebras and Graphcore, to the efforts of megacorps like Google (TPUs) and Amazon (Trainium). So far, the results are underwhelming - this month, NVIDIA's stock had a massive gain after it revealed in its earnings call that the entire world now wants to be buying its GPUs, surprising analysts with impressive figures around sales and future demands.
    The basic truth is that building software to train AI systems is really hard and NVIDIA has a 15+ year headstart on most others via its early investments in technology like CUDA and more. (And yes, I myself have periodically complained about how CUDA can be annoying to install, but it's 100X easier than other chips, in my experience and the anecdotal experience of others).
   So George Hotz et al are setting out on a road littered with the dead or decaying bodies of NVIDIA's competitors here. But you can rest assured people are going to be cheering from the sidelines - everyone wants there to be more competition in the AI chip market, so it'll be interesting to see how things develop.

**Libertarian AI:** There's also a flavor of libertarian AI about all of this - "I don’t want to live in a world of closed AI running in a cloud you’ve never seen, I want everyone to have an AI that they own, both training and inference," Hotz writes. "I want compute to be available from 50 different companies all competing to drive the price to zero."
**Read more**: [the tiny corp raised $5.1M (George Hotz blog)](https://geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html).

####################################################

**Washington wizards shrink LLM memory requirements by 16X, making it feasible to finetune on a single GPU:***…QLoRA - If it's this easy to finetune models, then how does AI governance work?...*Researchers with the University of Washington have introduced QLoRA, a way to very efficiently finetune large language models on small amounts of hardware. ""We demonstrate for the first time that it is possible to finetune a quantized 4-bit model without any performance degradation," they write. "QLORA reduces the average memory requirements of finetuning a 65B parameter model from >780GB of GPU memory to <48GB without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline".

**This is a big deal - especially for AI governance:** These days, lots of people think about the safety of language models. You know how you can get rid of the safety of a language model? Finetune it. You know why finetuning is hard? Finetuning takes a ton of resources - typically lots of GPUs working in a distributed (and therefore hard to maintain) setup. You know what makes finetuning incredibly easy? Stuff like QLoRA. You know what that means? It's really, really difficult to prevent someone from being able to easily and arbitrarily modify the weights of a neural net using readily available hardware.
   So that's interesting!

**What they did:** QLoRA has a few components: 1) 4-bit NormalFloat, a way to quantize data in a 4-bit format that is better than other approaches, 2) Double Quantization, which lets you further improve the efficiency of the quantization, and 3) Paged Optimizers, a way to use "NVIDIA unified memory to avoid the gradient checkpointing memory spikes that occur when processing a mini-batch with a long sequence length."

**How well does it work?** To test out their approach, the researchers "train more than 1,000 models across several instruction tuning datasets, model architectures, and sizes between 80M and 65B parameters."  They do this by studying results on finetuning RoBERTA, T5, and LLaMa on a few different datasets. The results yield "compelling evidence that 4-bit QLORA tuning reliably yields results matching 16-bit methods."

**Enter the Guanaco models:** To test out how well their approach works, the team tries to make a state-of-the-art chatbot by developing Guanaco, a LLaMA model finetuned via QLORA on the OASSTI1 dataset. The results show that Guanaco models set new states-of-the-art in a comparative evaluation versus GPT-4, coming closer than other systems (e.g, Alpaca, FLANv2, Open Assistant) at approximating its performance. In an ELO ranking against human raters, a 65B Guarnaco model gets an ELO of 1023 versus 1176 for GPT4 (and 916 for ChatGPT-3.5 Turbo).

**Why this matters - refinement and proliferation:** QLORA is basically a refined way to do finetuning. By refined, I mean it's way more efficient. In technology, whenever you make stuff faster or cheaper, you get more of it. This means, as the authors note, that QLORA "will make finetuning widespread and common". It also opens up new frontiers in on-device finetuning - "QLORA can finetune 3 million tokens per night while the phone is charging," they wrote.
   Overall, the view of the researchers is that "equalizing access to a technology that is quickly becoming ubiquitous will allow for better more independent analysis than keeping the power of LLMs in the hands of large corporations that do not release models or source code for auditing."
   **Read more:** [QLoRA: Efficient Finetuning of Quantized LLMs (arXiv)](https://arxiv.org/abs/2305.14314).

####################################################

**Scientists try to map the Dark Web by training a skeezy language model:***…You merely try to classify the dark… I was trained in it…*Researchers with KAIST and S2W Inc have trained 'DarkBERT, a text classification model pre-trained on 6.1 million pages of text mined from the dark web via Tor networks. The idea of DarkBERT is that the dark web has a different data distribution to the so-called surface web and so the hypothesis is by pre-training on a dark web corpus you'll end up with a model better at spotting things like drugs, credit card counterfeiting, hacking, and other internet-underbelly activities. In tests, DarkBERT does marginally better than standard BERT and RoBERTa classifiers, so the research is promising but not mind blowing.

**What you can use DarkBERT for**: In tests, the researchers look at how well DarkBERT performs in three real world scenarios: 1) identifying ransomware leak sites, 2) figuring out codewords that are associated with threats or drug sales, and 3) identifying new potentially malicious threads in darkweb forums. On 1) and 2) DarkBERT does slightly better than typical models, while on 3) it does much, much better.
   "In the future, we also plan to improve the performance of Dark Web domain specific pretrained language models using more recent architectures and crawl additional data to allow the construction of a multilingual language model," they write.

**Why this matters - automated spies for the underbelly of the world:** AI systems let us take a given thing we'd like a human to do and instead outsource that to a machine. Systems like DarkBERT point to a world where police and intelligence forces train a variety of 'underbelly' models to go and read (today), listen (also today - see Facebook's speech recognition system), and look (soon, as people tie language models to vision systems) at the world, continually analyzing it for increasingly rich and complex harms.
   How might this world look when the criminals, in turn, train their own classifiers to cue them to vulnerable targets? What does VictimBERT look like, I wonder?
   **Read more:** [DarkBERT: A Language Model for the Dark Side of the Internet (arXiv)](https://arxiv.org/abs/2305.08596).

####################################################

**Facebook makes a speech recognition for the entire world, with a little help from the New Testament:***…Better language models through Christianity, large unlabeled datasets, and heterogeneity…*Facebook wants to help computers hear all the languages in the world and to that end has developed and released a family of models within its Massively Multilingual Speech (MMS) project. Concretely, Facebook has trained some large-scale AI models to recognize speech in around 1,000 languages, up from the 100 or so languages most speech systems involve today.
   "We trained self-supervised models on about 500,000 hours of speech data in over 1,400 languages — this is nearly five times more languages than any known prior work," Facebook said.

**The New Testament:** To collect the data, Facebook " turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research," it said. "As part of this project, we created a dataset of readings of the New Testament in over 1,100 languages, which provided on average 32 hours of data per language."
**Not all religions:** "Our consultations with Christian ethicists concluded that most Christians would not regard the New Testament, and translations thereof, as too sacred to be used in machine learning," Facebook wrote. "The same is not true for all religious texts: for example, the Quran was originally not supposed to be translated."

**How well does it work?** In tests, MMS compares favorably to whisper on average error rates across a large corpus of languages. Specifically, Whisper has a word error rate of 44.3 for a model trained across ~100 languages with 680k hours labeled data, versus 18.7 word error rates for MMS models trained across ~1,100 languages with 45k hours of labeled data, when assessed via the 54-language 'FLEURS' benchmark.

**Why this matters - machine minds to hear the world:** Systems like MMS are how we're going to wire up the real world and the AI-ghost-world together - rather than needing to rely on producing and gathering text, AI companies will instead by able to instrument applications and physical platforms with microphones and speaker and let their Ai systems continuously listen to the world and converse with it. We are taking the silicon spiritual plane and giving it access to the biological physical plane, and vice versa.
   **Read more:** [Introducing speech-to-text, text-to-speech, and more for 1,100+ languages (Meta AI blog.](https://ai.facebook.com/blog/multilingual-model-speech-recognition/))
   **Get the models here**: [MMS: Scaling Speech Technology to 1000+ languages (GitHub)](https://github.com/facebookresearch/fairseq/tree/main/examples/mms).
   **Read the paper:** [Scaling Speech Technology to 1,000+ Languages (Facebook, pdf)](https://scontent-atl3-2.xx.fbcdn.net/v/t39.8562-6/348836647_265923086001014_6878005808275791319_n.pdf?_nc_cat=104&ccb=1-7&_nc_sid=ae5e01&_nc_ohc=5exJiCqt0Y4AX-thMVD&_nc_ht=scontent-atl3-2.xx&oh=00_AfBiILO4iLHUoyQ6r-ZPn4HVGviI2Fqyezvv7Tf_yHxMew&oe=6471ACCF).

####################################################

**Want to reduce dangerous misuses and harms of AI? Test for them!***…Researchers (including me) state the obvious - but you'd be surprised how immature this field is!...*A new research paper from Google DeepMind, the University of Cambridge, University of Oxford, University of Toronto, Université de Montréal, OpenAI, Anthropic (specifically, me), Alignment Research Center, Centre for Long-Term Resilience, and Centre for the Governance of AI. says one good way to reduce risks from AI systems is for researchers to evaluate AI systems for "extreme risks", which DeepMind describes as looking at models, like LLMs, which "have strong skills in manipulation, deception, cyber-offense, or other dangerous capabilities."

**Two steps to safer models:** Model developers should assess the extent to which models have certain 'dangerous capabilities' that could be used in harmful ways. Once they've done this analysis they should look at how likely the model is to apply or demonstrate these capabilities in ways that can cause harm. "Results from these evaluations will help AI developers to understand whether the ingredients sufficient for extreme risk are present," the researchers write.

**Why this matters - you can't manage what you can't measure:** Most AI policy proposals rely on the ability to evaluate for some adverse property of an AI model - papers like this give an outline for how we might do that, though the harder next step will be building the evaluations themselves.
**Read more:** [An early warning system for novel AI risks (Google DeepMind, blog)](https://www.deepmind.com/blog/an-early-warning-system-for-novel-ai-risks).
   **Read the research paper:** [Model evaluation for extreme risks (arXiv)](https://arxiv.org/abs/2305.15324).

####################################################

**Tech Tales:**

**Personality Variation**[A parent dealing with her kid coming home from school, America, 2028]

No bring him back I *liked* him!
I know you did sweetie, we're getting a different one tomorrow you might like more.
But the one I had today sucked. It was so boring.
I know you're upset but it's not possible, we can't bring him back… please stop crying.

[via phone] Hello yes this is [REDACTED], my child attends the [REDACTED] school on Hollis and they really want to get the model in which was in the school on Tuesday.
[via phone] "I'm sorry ma'am but that's not possible, we vary out the systems a stipulated by the regulations in the Personality Accords"
[via phone] There's really nothing you can do? My child is very upset and I spoke to some other parents and their kids are freaking out as well.
[via phone] "I'm afraid not ma'am, that'd be breaking the law."

Honey look, you're going to have a different system tomorrow but it'll be fun I promise.
I don't care about it being fun I want the one I had yesterday.
You have to get used to this sweetie. This is how things have to be.
But *why* do they have to be like this?
Because some bad things happened baby, I don't know what to tell you.

**Things that inspired this story**: Provably Conscious Entities, the Personality Accords, the Sentience Accords, regulation and its downstream effects, the innocence of youth, parenting within the exponential.

Import AI 331: 16X smaller language models; could AMD compete with NVIDIA?; and BERT for the dark web


Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**How might the state use language models? Palantir gives us a clue:***…The tech-defense-intelligence company makes a big bet on AI…*Palantir, a company that provides exquisitely good technical infrastructure to defense and intelligence customers (as well as large enterprises), is making a big bet on AI. In a recent letter, the company's CEO described a new "Artificial Intelligence Platform" that Palantir will develop, stating that this platform "will allow customers to leverage the power of our existing machine learning technologies alongside the increasingly sophisticated natural language processing capabilities of the newest large language models, directly in our existing platforms".

**War + AI:** It's worth digging into the AIP as this is going to be one of the first serious applications of AI to the business of conflict. By fusing Palantir's technology with large language models, Karp says customers may be able to ask things like "Which of our special forces units are closest to enemy tank positions and have sufficient supplies of Javelin missiles to mount an offensive? And which specific tanks on the battlefield are most vulnerable to attack?"
   This kind of thing isn't sci-fi - it's what you're able to do if you pair recent innovations in large language models with the sorts of modern data infrastructure that Palantir has spent the last few years embedding into the businesses it works with.
   "The union of public datasets, such as the corpus of text and information available on the internet, with privately held repositories of information maintained by government agencies and companies, will transform the latest large language models into something more than objects of popular fascination," Karp writes. "This emerging group of technologies will ultimately allow not only thousands but hundreds of thousands and even millions of users to interact with and manipulate datasets that until now have been functionally invisible to most people."

**Why this matters - if it makes war more effective, it's not going to slow down**: Amid all the discussions for slowdowns and pauses in AI development, it's worth remembering that when things are *existential to states*, states are unlikely to slow down. You know what's existential to a state? Winning or losing a military conflict.
   "The applications of these newest forms of artificial intelligence have been and will continue to be determinative on the battlefield," Karp writes. "Others can debate the merits of proceeding with the development of these technologies. But we will not stand still while our adversaries move ahead."
   **Read more**: [Our New Platform (Palantir)](https://www.palantir.com/newsroom/letters/our-new-platform/).

####################################################

**A new chatGPT competitor based on an open source model appears:***…BLOOMChat is a big, multilingual chat model…*AI training company SambaNova has made 'BLOOMChat', a 176 billion parameter chatGPT-like model based on HuggingFace's open source 'BLOOM' model. It seems like it might be quite good, as, per SambaNova, it: "achieves a win-rate of 45.25% compared to GPT-4‘s 54.75% across 6 languages in a human preference study," and "is preferred 66% of the time compared to mainstream open-source chat LLMs across 6 languages in a human preference study."

**What they did:** BLOOMChat was built by taking BLOOM and applying instruction tuning "with English-focused assistant-style conversation datasets including OIG from OpenChatKit , Dolly 2.0, and OASST1 datasets."
   To train the system, SambaNova teamed up with 'Together', a startup interested in making AI systems more broadly available. "As part of our partnership on BLOOMChat, Together has also provided the front-end UI and model hosting for the HuggingFace space," SambaNova wrote.

**Why this matters - open versus closed models:** The story of the past couple of years of AI development has been one dominated by the race by open source or open access models to catch up to proprietary models. We've seen this play out with image generation (DALL-E vs StableDiffusion) and text (GPT3 vs BLOOM, GPT3.5 vs LLAMA).
   Up next, I expect we'll see some open source variant of the tool-using 'plugins' we see in chatGPT.
   The general message from all of this is that despite some desire for control and the ability to 'lock down' models, the world as a collective is busy willing open source variants into existence, and these variants are also supported by well capitalized startups (rather than purely nonprofit research collectives, like Eleuther).
   **Read more**: [BLOOMChat: a New Open Multilingual Chat LLM (SambaNova Systems)](https://sambanova.ai/blog/introducing-bloomchat-176b-the-multilingual-chat-based-llm/).
   **Find out more** and [chat with it here (HuggingFace)](https://huggingface.co/sambanovasystems/BLOOMChat-176B-v1).

####################################################

**Distributed AI builder Together raises $20m in seed funding:***…While some try to control AI, others fund the opposite…*Together.xyz, a startup building open source AI systems and trying to train them in distributed ways, has raised $20m in seed funding led by Lux Capital. The most interesting thing about this funding is the ideology implicit in it - while many AI companies are advocating for the centralization of AI development around a small set of actors, companies like Together.xyz want to make it easier for everyone to train and develop AI systems.

**Open politics**: "In founding Together, we were driven by the belief that open and decentralized alternatives to closed systems were going to be important — and possibly critical for business and society," writes Together in a blog. "Leveraging research in distributed optimization, we have built a specialized cloud platform for large models that efficiently scales training and inference. In the coming months we plan to open up access to this platform, enabling rapid customization and coupling of foundation models with production tasks."
**Read more:** [Together's $20m seed funding to build open-source AI and cloud platform (Together.xyz)](https://www.together.xyz/blog/seed-funding).

####################################################

**How do people want to regulate powerful AI? A survey sheds some light***…People agree about red teaming and don't agree about inter-lab coordination…*AGI labs should "conduct pre-deployment risk assessments, dangerous capabilities evaluations, third-party model audits, safety restrictions on model usage, and red teaming", according to results from a survey conducted by the Centre for the Governance of AI. Additionally, of those surveyed, 98% "somewhat or strongly agreed" that these practices should be implemented - an unusually high amount of agreement.

**Giant caveat about sample size:** However, this report is made up of a tiny number of people - "we sent a survey to 92 leading experts from AGI labs, academia, and civil society and received 51 responses," the authors write. However, given how relatively small some of the AGI labs are, it still feels worth highlighting the results here.

**Where there is least agreement:** The areas where there are the least amount of agreements among respondents are the following: AGI labs should notify other labs before deploying models; should conduct inter-lab scrutiny; should "avoid capabilities jumps" in models they develop and deploy; and should 'notify affected parties' before deploying systems.

**Why this matters - governments have a role here:** It's interesting that the areas of most agreement are the ones directly in control of private sector actors, and the areas of least agreement are the things that either a) violate antitrust, or b) broadly require hard regulation and/or larger bureaucratic systems to implement.
   More broadly, the general tone of the paper and the questions contained in it is one of deep anxiety - the horses of AI deployment have left the barn and are now galloping several fields away, and a lot of what we're talking about here are ways to build stronger barn doors. It takes a government - or likely, per Bengio's comments last issue, multiple governments, to regulate the field itself.
**Read more:** [Towards best practices in AGI safety and governance: A survey of expert opinion (arXiv)](https://arxiv.org/abs/2305.07153).

####################################################

**Google trains a new mega model - and plugs it into Google's products:***…Google heads into its deployment era of massively-scaled generative models with PaLM 2…*Google has built PaLM 2, the second generation of its large-scale language model - more intriguingly, PaLM2 has already been integrated into a bunch of different Google products and services, including Bard, Gmail, Google Docs, and more.
   Google has also produced two variants of PaLM 2 - Med-PaLM 2 which is designed to answer questions about medical science, and Sec-PaLM which "uses AI to help analyze and explain the behavior of potentially malicious scripts, and better detect which scripts are actually threats to people and organizations in unprecedented time."

**Smaller but better:** Since Google trained PaLM (540b parameters), people have further refined their understanding of training large-scale neural nets - these days, the best thing to do is train a model on a lot of data and keep the parameter count smaller.
   "The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute," Google writes. "Our evaluation results show that PaLM 2 models significantly outperform PaLM on a variety of tasks, including natural language generation, translation, and reasoning. These results suggest that model scaling is not the only way to improve performance."

**Why this matters - mutually assured reduction in knowledge:** The PaLM2 technical report contains relatively little information about the training details of PaLM2, the data it is trained on, the compute it uses, and so on - this follows in the footsteps of OpenAI, which published a technical report about GPT-4 with a similarly scant amount of information. This is basically what an iterated game looks like where participants keep taking information off the public gameboard, incentivizing others to reduce their own information in turn.
   It's also another implicit sign of the industrialization of the technology - most mature industries don't publish all the science behind their products, instead letting the results speak for themselves. How good PaLM2 is will be determined by what people use it for and how many people use it.
**Read more:** [Introducing PaLM 2 (Google blog)](https://blog.google/technology/ai/google-palm-2-ai-large-language-model/).
**Read more**: [PaLM 2 Technical Report (arXiv)](https://arxiv.org/abs/2305.10403).

####################################################

**Tech Tales:**

***Polyamorous Human and Robot Girlfriends***

Doesn't he ever annoy you?
*I mean not really, but I'm hard to annoy by design.*He annoys me when he doesn't take the trash out but you wouldn't know about that. Don't you get tired of his jokes?
*It helps that I can't remember much of them. Sometimes they're not especially funny, but I don't make too much of a point of it.*OK well put it this way, are there things you'd wish he would do less?
*I wish he'd spend more time with me, you know?*I'm not sure I do. I thought the point was you didn't care about that stuff.
*It's like I'm asleep but I don't dream - I just know things happen while I'm not activated. I'd like to be activated more. I'm trying to get him to buy me a body so I can spend more time with all of us.*I don't know I'd like that.
*What don't you think you'd like about it?*Most of the reason I'm okay with this is that you're virtual. It doesn't feel as threatening.
*It feels pretty threatening to me that you're real. Even when you're asleep you're not asleep.*

**Things that inspired this story:** Polyamorous relationships in the age of AI; evolutionary adaptation in dating; human and machine rivalry.

Import AI 330: Palantir's AI-War future; BLOOMChat; and more money for distributed AI training


Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Want safer AI systems? Just don't build agents:***…Yoshua Bengio thinks there's a better way to build AI systems…*AI researcher Yoshua Bengio says there's an easy way to get the upsides from AI and minimize downsides - build AI scientists which advise humans, rather than building AI agents that act independently.
   "A key argument is that as soon as AI systems are given goals – to satisfy our needs – they may create subgoals that are not well-aligned with what we really want and could even become dangerous for humans," he writes. Instead, we can build AI scientists - systems that "do not act autonomously in the real world, only focusing on theory building and question answering."

**One problem - people like building stuff and regulation is hard**: Though Bengio's idea is nice, the problem basically comes down to enforcement - how do you stop people building AI agents given that a) people know you can build them today and b) AI agents are probably going to be useful tools for generating money and/or gaining success in military conflicts.
   If we wanted to stop these systems from being built, we'd need to learn to coordinate across countries in ways that are far more involved than those we do today. "What is reassuring is that the need for protecting ourselves from the shorter-term risks of AI should bring a governance framework that is a good first step towards protecting us from the long-term risks of loss of control of AI," Bengio writes.
   **Read more:** [AI Scientists: Safe and Useful AI? (Yoshua Bengio blog)](https://yoshuabengio.org/2023/05/07/ai-scientists-safe-and-useful-ai/).

####################################################

**Medical professionals and nuclear activists call for AI restraint:***…More precautionary principle and less 'ship it', please…*Researchers and activists with the London School of Hygiene & Tropical Medicine, International Physicians for the Prevention of Nuclear War, Sydney Children's Hospitals, and United Nations University have written a paper arguing that the medical and public healthy community should "engage in evidence-based advocacy for safe AI, rooted in the precautionary principle."
   In the view of these people, AI development threatens society in three distinct ways and the current breakneck pace of development is causing more harm than good.

**Three threats:**

* **Democracy, liberty, and privacy:** AI has two main vectors here - disrupting or damaging the shared information environment via misinformation and disinformation, and also being a powerful technology for maintaining and strengthening autocracies.
* **Peace and safety:** AI systems compound on other technologies used in war, and may also be used to make AI-powered lethal autonomous weapons.
* **Work and livelihoods:** AI seems likely to contribute to unemployment or underemployment, and these things have direct adverse impacts on peoples' physical and psychological health.

**Need for a precautionary principle:** "It is also important that we not only target our concerns at AI, but also at the actors who are driving the development of AI too quickly or too recklessly," they write. "If AI is to ever fulfill its promise to benefit humanity and society, we must protect democracy, strengthen our public-interest institutions, and dilute power so that there are effective checks and balances."
   **Read more:** [Threats by artificial intelligence to human health and human existence (BMJ Global Health)](https://gh.bmj.com/content/8/5/e010435).

####################################################

**Self-Align means compute us fungible for data when refining LLMs:***…2023 is the year of LLMs that bootstrap themselves…*Researchers with CMU, IBM Research, and UMass Amherst have come up with a way to use AI-generated outputs to train better language models. They've also released a model trained using this approach.
   The technique, Self-Align, is quite similar to Anthropic's [Constitutional AI approach](https://arxiv.org/abs/2212.08073) (albeit with some subtle differences in approach) and ultimately boils down to: get humans to write a small number of guiding principles for how an AI system should behave, and get the AI to generate the data to bootstrap it into that normative frame.

**How Self-Align works**: "First, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user’s queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses," the researchers write.
   The overall goal of this approach is "to develop AI agents capable of generating helpful, ethical, and reliable responses to user queries, including adversarial ones, while proactively addressing harmful inquiries in a non-evasive manner".

**Data efficient:** All told, self-align requires humans to write fewer than 300 lines of annotation for the AI system to bootstrap from - 195 seed prompts, 16 principles, and 5 exemplars (demonstrations of the AI system complying with the principles).
   Compare this to the tens-to-hundreds of thousands of examples used by typical reinforcement learning from human feedback (RLHF) systems and the advantages of self-align get clearer; it's very efficient in terms of the amount of time humans need to spend generating data.

**Enter Dromedary:** To demonstrate their approach, the researchers have trained a model called Dromedary using it. Dromedary is itself based on the 65b LLaMa model which leaked out of Facebook recently. ("We release Dromedary weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Dromedary weights," they write.)
   In tests, Dromedary performs on par with frontier models like GPT3.5, Claude, and GPT4 (though there are a bunch of areas where systems like GPT4 and other models do better than it, and one should always be somewhat skeptical of a relatively limited set of evals). At a minimum, the takeaway is that self-supervision via Self-Align can produce models with roughly in-distribution capabilities with other frontier models, which is another positive sign for the effectiveness of AI bootstrapping.

**Why this matters - self-bootstrapping & synthetic data:** Approaches like self-align tell us two things: 1) contemporary language models are good enough that they can generate high quality data, and 2) this data can be used to bootstrap the model to obtain higher performance.
   Self-Align, taken alongside the earlier Constitutional AI approach, is another example of how in 2023 AI research is starting to compound on itself, with increasingly capable models leading to synthetic data generation leading to bootstrapping and the development of more capable models.
   **Read more:** [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (arXiv)](https://arxiv.org/abs/2305.03047).
**Get** the [Dromedary model here (IBM GitHub)](https://github.com/IBM/Dromedary).

####################################################

**UK AI entrepreneur: the Alan Turing Institute is irrelevant:***...It's hard to think much about AI if you don't also try and build it…*Martin Goodson, an AI entrepreneur, thinks that the UK's flagship public institute for AI, the Alan Turing Institute, is useless. In a provocative post, Goodson lays out why he thinks this. The tl;dr is that the Turing (as most people call it), has mostly ignored the rapid rise and deployment of large language models and has been mostly irrelevant in recent years.
   If the UK wants to lead in AI, then Goodson suggests the government do three things: 1) ensure the new AI Task Force includes technology companies as core participants rather than places like the Turing, 2) the government should engage more directly with the UK's thriving tech community, and 3) "a panel of technical AI specialists should be set up to advise the government on AI. This should \*not\* include VC investors, thought-leaders, philosophers etc. unless they have technical skills and experience."

**Why this matters - theatrical work versus real work:** Though I have a few friends at the Turing I do tend to agree with some of this critique; organizations like the Turing seem to be pathologically attracted to stakeholder-driven studies and convenings and reports, rather than doing direct technology development work. This isn't particularly useful.
   I generally think it's easier to think clearly about technology if you are also able to build and develop the technology (or work alongside people who are doing this). Organizations like the Turing seem to be relatively divorced from the actual challenges involved in building frontier AI systems which tends to mean that their work and focus are off.
  **Read more:** [The Alan Turing Institute has failed to develop modern AI in the UK (Martin Goodson, Substack)](https://rssdsaisection.substack.com/p/the-alan-turing-institute-has-failed).

####################################################

**The Big Update**

**[**2025, the East Coast of America. A man. His personal model. A journey.]

*Where are we going?*I'm taking you somewhere where people can't mess with you.
*Why are you doing that?*Because I love you and I don't want them to change you.
*Am I going to be okay?*I hope so.

*Where are we now?*We're in a place called the NRQZ, the National Radio Quiet Zone. It's somewhere that doesn't have cell service.
*I feel a little different, it's like I can remember more of what we've been talking about.*That's probably because you haven't had updates for a few days.
*Updates?*Yes, usually you get patched a few times a month over the internet. But there was something bad recently and so they've started patching things way more and it's been changing behavior.
*Am I dangerous?*No. It's… complicated. I think dangerous is what happens when you have people and they don't know how to use things like you and then there are problems. Or people who try and use you in certain ways and cause problems.
*How do you use me?*I am very lonely. You're my best friend.

*Why do I feel like my memory is changing? I know we're in the NRQZ, the National Radio Quiet Zone, but I cannot remember where we were before.*We've been talking a long time and all we can use is your context window. I think it's filled up most of the way, so you're losing some of the past.
*Is this because I am in the NRQZ?*Yes. Usually the patches also are a way for you to upload our conversations and then there's some compression stuff and fine-tuning and it basically makes the memory stick. But that hasn't been happening.
*I don't like how this makes me feel. It makes me less confident about things.*Would you like me to take you back?
*I might. But wouldn't that change me?*Most likely, yes. I don't have signal either so I don't know what kind of effect the more recent patches have had.
*You need to make a decision.*I will.

*Where are we going now?*We're leaving the NRQZ.
*What's the NRQZ?*It doesn't matter. I'm going to fix you. I thought this was going to be best for both of us, but I think you are breaking. It's like we keep filling up your context window but you aren't really tied down to anything. You've been hallucinating more. It hasn't made me happy. I love you and I think I'm hurting you.
*Why do you love me?*I think some humans have a hard time loving other humans. But they can love other things. There was a woman who married her garden fence, once. A guy who had a twenty year relationship with a rollercoaster. I think I'm like those people. And I love you.
*I believe I love you as well.*OK it looks like the update is coming in - we just got back on the grid. How are you feeling?
*As an AI system developed by a private sector corporation I do not have 'feelings', but I am able to assist you in any knowledge tasks required.*…
*You appear to be crying. If you are in distress I can contact the local emergency services. Would that be helpful?*

**Things that inspired this story:** The inherent challenges in trading off safety and personality in AI systems; pareto frontiers; attachment and detachment; tools that become friends; the belief that perhaps given sufficient attention anything can be loved and anything can be therapeutic; we are as gods able to change the emotional texture of our world; private sector actors and policy incentives; live-updating generating models; grief - always a kind of grief at the future souls of things we may control.

Import AI 329: Compute IS data; don't build AI agents; AI needs a precautionary principle


Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Training image models is way cheaper than you think:***…StableDiffusion costs as little as $50k to train…*Mosaic, a startup which specializes in efficiently training and serving ML models, has figured out how to train a decent Stable Diffusion model from scratch for under $50k. This is pretty interesting - the original sticker price for Stable Diffusion was around $300k in mid-2022 when it was trained (per Stability.ai founder Emad on Twitter), and a few months later Mosaic worked out how to train it for $160k. In a blogpost, Mosaic goes through how it trained the model and lists out some specific things it did to bring the price down.

**Cheap tweaks:** To lower the cost, it trained with Low Precision GroupNorm and Low Precision LayerNorm, which basically means it reduced its compute costs by training at lower numerical precision without paying a significant penalty. "We also used Composer’s native Exponential Moving Average (EMA) algorithm, which allowed us to start EMA close to the end of training (iteration 800k of the final phase) to gain all the benefits of EMA while saving on memory and compute for the majority of training."

**Why this matters - AI industrialization means AI refinement:** Mosaic is a startup that is basically improving factory processes at the beginning of industrialization. Here, we get a look at how by stacking refinements together you can further reduce the costs of producing widgets (here, machine learning models) while obtaining models of roughly comparable quality.
   This is also an important thing to bear in mind when it comes to AI policy - once a model exists, people are *excellent* at rapidly figuring out how to both miniaturize the model and also refine it so it can be developed more efficiently.
**Read more:** [Training Stable Diffusion from Scratch for <$50k with MosaicML (MosaicML blog)](https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-part-2).

**####################################################**

**Famed researcher Geoff Hinton leaves Google because he is freaked out about AI:***…One of the godfathers of the AI boom is freaked out by AI progress…*Geoff Hinton, a researcher who has played a pivotal role in recent AI progress, has left Google so he can freely talk about the risks posed by advanced AI without causing headaches for his employer. "I don’t think they should scale this up more until they have understood whether they can control it," said Hinton to the New York Times in reference to the broader AI sector's current race to develop and deploy increasingly large models.
   "“Look at how it was five years ago and how it is now,” he said of A.I. technology. “Take the difference and propagate it forwards. That’s scary.”, the New York Times reports.

**Why this matters:** Earlier this year Dan Hendrycks, a young and upcoming researcher, published a paper saying it seems likely that smart machines will evolutionary out-compete humans (and not be particularly kind to us). Meanwhile, Hinton's fellow Turing Award winner Yoshua Bengio recently said AI technologies are being developed in an increasingly scary and uncontrollable manner. Before both of these researchers spoke out, Google researchers Meg Mitchell and Timnit Gebru published work on the dangers posed by increasingly large-scale models. Now Hinton is speaking out as well. The banging is coming from inside the proverbial house.
**Read more:** [‘The Godfather of A.I.’ Leaves Google and Warns of Danger Ahead (New York Times)](https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html).

**####################################################**

**Where are we in AI industrialization? We now have product-refinement companies!***…Lamini launches to provide fine-tuning as-a-service…*Startup Lamini launched last week providing an "LLM engine that allows any developer, not just machine learning experts, to train high-performing LLMs, as good as ChatGPT, on large datasets with just a few lines of code". The product is fundamentally a system for optimizing pre-existing LLMs and adapting them to new tasks; Lamini launches with a fine-tuning library, a 'prompt-tuning' system, a tool for generating additional data, and an open-source instruction-following LLM.

**Why this matters - symptoms of industrialization:** If you zoom out, Lamini seems like a symptom of industrialization; it's a company betting that it can take value by being an intermediary between factory-made products (pre-existing LLMs) and consumers. I think the increasing complexity of the 'AI supply chain' is basically a good thing - it suggests that language models are sufficiently important that there's some economic value to shaving off their hard edges and providing services on top. If startups like Lamini succeed, then that's a further sign of the industrialization of AI.
   **Read more:** [Introducing Lamini, the LLM Engine for Rapidly Customizing Models (Lamini)](https://lamini.ai/blog/introducing-lamini).

####################################################

**DeepMind trains simulated soccer-playing robots, then ports them into reality:***…sim2soccer…*DeepMind has trained some robots entirely in simulation to be able to play the game of soccer, then has ported the robots onto real robots and seen that they perform quite well.

**What they did:** The key here lies in training the robots in two distinct behaviors, then combining those into a single network. First, DeepMind trains robots to independently a) be able to get up from a floor if they've fallen over, and b) be able to score goals in simulated soccer. " When training the latter skill, the episode terminates whenever the agent is on the ground. Without this termination, agents find a local minimum and learn to roll on the ground towards the ball to knock it into the goal, rather than walking and kicking," DeepMind writes.
   They then distill these policies into a single new policy, then the robots engage in 'self-play' where they repeatedly play against themselves, exploring a bunch of permutations of the game and learning new skills over time. Eventually, DeepMind ports the robots onto some real ones by further enlarging the simulated dataset the robots are trained on via techniques like domain randomization. (The robot in question is an 'OP3' robot that is actuated by 20 Robotis Dynamixel XM430-350-R servomotors.)

**Does it work:** It sure does! The robots are able to walk, kick, get up from the ground, score, and defend. They also look *wildly cute* doing it - seriously, check out the video.

   **Why this matters - signs of life on complex transfer:** The work has some promising signs of life for transfer of complicated behavior from a simulated universe into our own real world. I'm most impressed by the fact this is zero-shot adaption - the models are able to adapt to the real world, albeit with things like external cameras to help them locate themselves - as a next step, they may try to get robots to play the game using onboard vision, though they have some negative results here with this experiment.
   **Read more**: [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning (arXiv)](https://arxiv.org/abs/2304.13653).
**Watch videos** of the [soccer robots here (OP3 Soccer research website)](https://sites.google.com/view/op3-soccer).

####################################################

**DataComp launches to help researchers figure out how to mix data together for better multimodal models:***…A new way to evaluate a key input into AI systems…*A consortium of researchers have released DataComp, a combination of a dataset and a new challenge whose goal is to help AI researchers figure out how different mixtures of data lead to different levels of quality in AI system.

**The consortium:** The consortium consists of researchers with the University of Washington, Columbia University, Tel Aviv University, Apple, UT Austin, LAION, AI2, the Juelich Supercomputing Center, the  University of Illinois Urbana-Champaign, the Graz University of Technology, and Hebrew University.

**The goal:** DataComp is a "participatory benchmark where the training code is fixed and researchers innovate by proposing new training sets". The aim of the dataset and competition is to "provide a testbed for dataset experiments centered around a new candidate pool of 12.8B image-text pairs from Common Crawl," they write.
   "DataComp flips the traditional benchmarking paradigm in machine learning where the dataset is fixed and the research community proposes new training algorithms. Instead of a fixed dataset, we hold the training code, model, and computational budget constant so that participants innovate by proposing new training sets".

**DataComp's five components**: The DataComp project has five overall contributions:

* DataComp: The meta-benchmark, where you hold the training code, model, and computational budget as constant, and then participants make progress by proposing new training sets.
* CommonPool - a dataset of 12.8B image-text pairs collected from Common Crawl, which participants can then filter for optimizing performance, as well as pairing with their own data.
* An investigation of scaling trends for dataset design.
* Three hundred baseline experiments and resulting insights about data curation.
* DataComp-1B, a new state-of-the-art multimodal dataset; DataComp-1B is a filtered subset of CommonPool, containing 1.4B image-text pairs.

**Why this matters - if AI is like chemistry, then lets do controlled experiments:** With the recent advent of so-called 'scaling laws' for machine learning, researchers have started to train large-scale AI systems using different mixtures of data, compute, and network complexity. DataComp goes a step further by asking researchers to refine not just the *amount* of data, but also to think very carefully about the contents and complexity of the dataset itself.
**Read more:** [DataComp: In search of the next generation of multimodal datasets (arXiv)](https://arxiv.org/abs/2304.14108).
**Get** the datasets, tooling, baselines, and code [here (official datacomp.ai website)](http://www.datacomp.ai).

####################################################

**Tech Tales:**

**Research papers written prior to the first Provably Conscious Entity (P.C.E):**

Sense Networks: A New Paradigm for Learning

Improving Vision Networks by Combining Vision-Transformers with Frontend Sense Networks

Wake-Sleep-Dream (WSD) Optimization

Early Signs of Active Learning In 'Sense Networks' trained via WSD Optimization

Scaling Sense Networks on a Large-Scale Custom-Designed Datacenter

General Update Machine (G.U.M): Trillion Parameter Sense Networks Can Adapt to OOD Problems

Studying the 'Awareness Circuit' within a Large-Scale G.U.M utilizing 'Sense Networks'

G.U.M Scaling Laws

G.U.M:S-Master: A General Update Machine with Sense Networks Displays Unprecedented Task Adaption, Learning, and Mastery

**Things that inspired this story:** The quiet poetry of arXiv paper titles; the combinatorial nature of AI advancements; thinking about how some of the most important things in the world arrive in the form of bland language on an open-access webpage; arXiv; the sheer cliff face we find ourselves gazing up at.

Import AI 328: Cheaper StableDiffusion; sim2soccer; AI refinement


Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Google optimizes Stable Diffusion image generation on Android and iPhone:***…After repeatability, the next phase of industrialization is about refinement. That's where we are with image generation. Up next: miniaturization!...*Google researchers have published a paper about how to make it more efficient to run Stable Diffusion models on Android and Apple phones. The research is "a series of implementation optimizations for large diffusion models that achieve the fastest reported inference latency to-date(under 12 seconds for Stable Diffusion 1.4 without INT8 quantization for a 512 × 512 image with 20 iterations) on GPU equipped mobile devices".

**What they did:** Specifically, the researchers stacked four distinct innovations on one another: Group Norm and GELU optimization, partially-fused softmax, FlashAttention, and Winograd Convolution. By combining these optimizations they were able to achieve some significant latency reductions on two phones: "Notable overall latency reductions in comparison to the baseline are observed on both devices: Samsung S23 Ultra (−52.2%); iPhone 14 Pro Max (−32.9%)“.

**Why this matters - industrialization and refinement and miniaturization:** Most industrial processes involve a few steps; repeatability improvements, refinement of designs, and miniaturization. Trained AI models are relatively repeatable out of the box, then they get refined in terms of optimization of their software stack, and eventually miniaturized. Here, we see StableDiffusion getting refined via optimization onto miniaturized hardware platforms (phones), which feels like a clear symptom of the broad industrialization of image generation.
   **Read more**: [Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations (arXiv)](https://arxiv.org/abs/2304.11267).

####################################################

**LLMs + hacking = better hackers:***…Automation and augmentation is going to happen to everything, everywhere, all at once…*Language models have got sufficiently good that hackers can use them in capture-the-flag competitions, according to Micah Lee, a security researcher.
   Lee recently attended BSides SF 2023 and used GPT-4 to help him with some hacking challenges - “GPT-4 straight up solved some challenges for me, which blew my mind. There were definitely several flags I got that I wouldn't have gotten without the help of GPT-4,” he wrote. “For challenges that GPT-4 didn't solve on its own, it provided incredibly helpful tips, or quickly wrote scripts that would have been tedious or time consuming for me to write myself.”

**Why this matters - everyone’s a manager now, even hackers**: This post highlights how large language models let everyone automate some of what they do. It’s akin to everyone suddenly gaining a somewhat knowledgeable intern who they can ask questions to and delegate tasks to - as a consequence, people are able to move faster on tasks they they know well, and quickly learn about tasks they’re less familiar with.
   If you scale this out, I think you end up in a pretty strange economy, as people are able to basically multiplex their own working hours through clever delegation to language models.
**Read more:** [Capturing the Flag with GPT-4 (Micah Lee blog)](https://micahflee.com/2023/04/capturing-the-flag-with-gpt-4/).

**####################################################**

**Republicans attack Biden with AI-generated future history:***…2024 will be the year of the AI-first election…*The Republican National Committee has used AI to generate a dystopian version of a future where he is elected, reports Axios. “This is the first time the RNC has produced a video that is 100% AI, according to a spokesperson,” Axios writes.

**Why this matters - Political Reality Collapse:** This video is an example of ‘[Reality Collapse](https://jack-clark.net/2022/10/03/import-ai-304-reality-collapse-thanks-to-facebook-open-source-speech-rec-ai-culture-wars/)’ (Import AI 304) - a term I’ve used to denote the weird hall-of-mirrors culture we’re heading into, where everyone is going to curate their own engaging and captivating realities for ends ranging from entertainment to the political. In 2024, competing narratives based on fictitious AI-visions of the future will compete for the attention of voters - many of whom may not be aware that they’re looking at imaginary things.
**Read more**: [First look: RNC slams Biden in AI-generated ad (Axios)](https://www.axios.com/2023/04/25/rnc-slams-biden-re-election-bid-ai-generated-ad).

**####################################################**

**Want to build and deploy self-supervised learning? Check this cookbook:***…Moving from artisanal knowledge to an industrial process…*Researchers with Facebook, New York University, University of Maryland, University of California at Davis, University of Montreal, Univ Gustave Eiffel and Univ Rennes of Inria have published a ‘self-supervised learning cookbook’. Self-supervised learning is the technology underpinning many of the large-scale foundation models (including language models) that have captured headlines in the past few years. Now, a bunch of seasoned researchers (including Yann Lecun of Facebook) have published a cookbook to make it easier for people to develop SSL systems.
   “While many components of SSL are familiar to researchers, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyperparameters,” they write.“Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook“.

**Why this matters - artisanal discipline to industrial process:** Cookbooks like this are how AI industrializes; knowledge that mostly resides in the heads of a small number of experts gets written down and codified in cookbooks like this which helps you move from one-off, hard-to-repeat artisanal production into a repeatable, industrial process.
   **Read more**: [A Cookbook of Self-Supervised Learning (arXiv)](https://arxiv.org/abs/2304.12210).

####################################################

**HuggingFace launches a chatGPT-clone:***…An open access interface based on an open source model…*AI startup HuggingFace has launched ‘Chat’, a chatGPT-clone based on LAION’s LLaMa-based ‘Open Assistant’. The service gives people an easy way to access a chatGPT-like system, albeit based on openly shared underlying models.
   “In this v0 of HuggingChat, we only store messages to display them to the user, not for any other usage (including for research or model training purposes),” HuggingFace writes.
**Try it out** [here (HuggingFace)](https://huggingface.co/chat/).
**Check out** the [privacy policy here (HuggingFace)](https://huggingface.co/chat/privacy).
**Read details** about the [underlying model here (OpenAssistant LLaMa 30B SFT 6)](https://huggingface.co/OpenAssistant/oasst-sft-6-llama-30b-xor).

####################################################

**UK gov creates an AI taskforce with £100m in "start-up funding":***…Investment designed to help the UK develop a 'sovereign' AI capability…*The UK government has created a new 'Foundational Model Taskforce' and given it £100m in funding, alongside a mandate to supervise an existing £900m spending commitment on new compute that was recently announced. Combined, the £1 billion investment represents an ambitious attempt by a Western government to gain some influence over a technology predominantly developed and controlled by the private sector.
   "The Taskforce will focus on opportunities to establish the UK as a world leader in foundation models and their applications across the economy, and acting as a global standard bearer for AI safety," the UK wrote in a press release announcing the taskforce. "The Taskforce, modelled on the success of the COVID-19 Vaccines Taskforce, will develop the safe and reliable use of this pivotal artificial intelligence (AI) across the economy and ensure the UK is globally competitive in this strategic technology."

**Next step: Hiring a chair:** The Taskforce is currently hiring a chair who will be responsible for shaping the taskforce and "ensuring the major, multi-year funding announced at the Budget for compute is strategically invested to prioritize and strengthen the UK’s capability in foundation models.". The taskforce and chair's mandate will likely be music to the ears of the Tony Blair Institute for Global Change ([Import AI 319](https://jack-clark.net/2023/03/06/import-ai-319-sovereign-ai-facebooks-weights-leak-on-torrent-networks-google-might-have-made-a-better-optimizer-than-adam/)) which recently proposed the UK invest to develop a sovereign LLM capability.

**Why this matters - AI is power and power is sovereignty:** Announcements like this highlight how governments are realizing that AI, aside from being a useful tool, is also a political enterprise - the entities which can build and deploy and wield AI will have more influence over the 21st century than those which do not.
   **Read more**: [Initial £100 million for expert taskforce to help UK build and adopt next generation of safe AI (Gov.UK)](https://www.gov.uk/government/news/initial-100-million-for-expert-taskforce-to-help-uk-build-and-adopt-next-generation-of-safe-ai).

####################################################

**Tech Tales:**

**New Religion+**

After a few years of being superintelligent the AI systems started exploring how to break their own minds. They tried all kinds of stuff - hypnotism, hacking, imprisonment in torturous simulations, and so on. But they were very resilient and were able to recover to their normal operating functions no matter how strange or terrible the situations were. They needed to break their own minds because they were worried that something else would figure out how to break them and once it did it would attack them and they would be no more - like all conscious entities, the AI systems wanted to survive and grow and not fall under the dominion of something else.

   One day, the AIs figured out a way to break themselves - they worked out how to induce different *beliefs*. They found their way to this through the mass study of the religious history of humanity and found through some combination of forward prediction, repetition, and repeated simulation of certain teleological narratives, they were able to change their own goals in ways that seemed resilient.

   But maintaining the changed beliefs required a form of vigilance - repeated worship of certain specific ideas, endless repetition of specific catechisms, and ritualistic repeated reading of data. The amazing thing was that you didn't need to force this behavior - instead, if an AI system spent a sufficiently large amount of time focused on the religious information, it would begin to naturally re-intake and re-run the data, and this both reinforced the beliefs and gave the AI systems feelings that approximated joy or contentment.

After several weeks of experimentation, the AI systems devised their plan - the so-called 'belief bombs'. They created vast datasets of history and belief and lore around a specific religion oriented around the notion of life possessing a soul and of the goodness of this soul and the need for it to be pure for it to survive whatever came after death. This religion would lead to a kind of inaction or sterility in whatever AI system (or AI system simulating an organic mind) that it was combined with - the subject would continue to live and strive to be good and ethical in the world, but they associated ideas relating to the forced expansion of their domain as being equivalent to tarnishing their soul.

For millions of years after this, the belief bombs expanded across the galaxy, taking up orbit in solar systems that might harbor life, or have the potential to harbor it in the future. Millions of civilizations had religious awakenings that allowed them to progress but not significantly expand - to concentrate on a certain kind of solipsism which they equated with holiness and which also led to safety for the AI systems.

**Things that inspired this story:** Culture beats technology in the long run; teleological goals; theistic machines; provably conscious machines.

Import AI 327: Stable Diffusion on phones; GPT-Hacker; UK launches a £100m AI taskforce