update

dist/index.html

@@ -108,7 +108,7 @@
         releases a new crawl containing 200 to 400 TiB of textual content obtained via automatic web crawling usually
         every 1 or 2 months. </p>
     <p>As an example, the latest CC crawl (April 2024) contains 2.7
-        billion web pages, totaling 386 TiB of uncompressed HTML text content<d-footnote>Note that the size changes from crawl to crawl</d-footnote>.
+        billion web pages, totaling 386 TiB of uncompressed HTML text content<d-footnote>Note that the size changes from crawl to crawl. Note also that we use "dump" or "crawl" interchangeability in this report.</d-footnote>.
       Ninety-six crawls have been released since 2013 and 3 crawls from 2008 to 2012, which are in a different (older) format.
       <d-footnote>We have not processed these 3 older crawls.</d-footnote> </p>
   
@@ -150,7 +150,7 @@
         scores.</p>
     <p>Our ablation models were trained using <a
             href="https://github.com/huggingface/nanotron"><code>nanotron</code></a> with this config [<strong>TODO:
-        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Our ablation models have 1.82B parameters (including embeddings), used the Llama
+        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Our "ablation models" have 1.82B parameters (including embeddings), used the Llama
         architecture with a 2048 sequence length, a global batch size of ~2 million tokens, and the GPT2 tokenizer. For most
         ablations we trained on ~28B tokens (roughly the Chinchilla<d-cite bibtex-key="hoffmann2022training"></d-cite> optimal training size for this
         model size). To confirm relative performance improvements after each step of filtering we conducted longer training runs on 350 billion tokens as mentioned further below.</p>
@@ -161,7 +161,7 @@
     <ul>
         <li>small variance between runs trained on different samplings of the same
             dataset: we want our runs on a subset of the data to be representative of the whole dataset, and the
-            resulting scores to have as little sensitivity to exact data choice as possible (apart from larger ablations that we are concerned with)
+            resulting scores to be, in the limit of what is possible, less sensitive to exact data point choices than to our filter ablations.
         </li>
     </ul>
     <ul>
@@ -204,10 +204,10 @@
         full page HTML and request metadata. <strong>WET</strong> (WARC Encapsulated Text) files provide a text only
         version of those websites.</p>
     <p>A large number of datasets take the WET files as their
-        starting point. In our experience the default text extraction (extracting the main text of a webpage from
-        its HTML) used to create these WET files is suboptimal and there are a variety of open-source libraries that
-        provide better text extraction (by, namely, keeping less boilerplate content/navigation menus). We extracted
-        the text content from the WARC files using the trafilatura library<d-cite bibtex-key="barbaresi-2021-trafilatura"></d-cite>, which from visual inspection of the results provided good quality extraction when compared to other libraries.</p><aside>You can also find a benchmark on text extraction libraries <a href="https://github.com/scrapinghub/article-extraction-benchmark/blob/master/README.rst">here</a>.</aside>
+        starting point. In our experience the default text extraction used by Common Crawl to create these WET files is suboptimal for the goals of LLM pretraining<d-footnote>In particular we suspect that it keeps too much boilerplate content and navigation menus.</d-footnote> and there are a variety of open-source libraries that
+        provide better text extraction. We extracted
+        the text content from the WARC files using the trafilatura library<d-cite bibtex-key="barbaresi-2021-trafilatura"></d-cite>, which from visual inspection of the results provided good quality extraction when compared to other libraries.</p>
+    <aside>You can find a benchmark comparing several text extraction libraries <a href="https://github.com/scrapinghub/article-extraction-benchmark/blob/master/README.rst">here</a>.</aside>
     <p>To validate this decision, we processed the 2019-18 dump
         directly using the WET files and with text extracted from WARC files using trafilatura<d-footnote>We used trafilatura default options with <code>favour_precision=True</code>.</d-footnote>. We applied the same
         processing to each one (our base filtering+minhash, detailed below) and trained two models. While the
@@ -221,10 +221,11 @@
         <figure><img src="assets/images/wet_comparison.png"/></figure>
         <div id="plot-wet_comparison"></div>
     </div>
+
     <h3>Base filtering</h3>
-    <p>Filtering is an important part of the curation process. It
-        removes part of the data (be it words, lines, or full documents) that would harm performance and is thus
-        deemed to be “lower quality”.</p>
+    <p>Filtering is an important part of the curation process. It consists in
+        removing part of the data (which can consists in removing words, lines, or even full documents) that lower the performances of the model and is thus
+        deemed to be “lower quality” in our eval-driven process of dataset crafting.</p>
     <p>As a basis for our filtering we used part of the setup
         from RefinedWeb<d-cite bibtex-key="penedo2023refinedweb"></d-cite>. Namely, we:</p>
     <ul>
@@ -243,26 +244,25 @@
         </li>
     </ul>
     <p>After applying this filtering to each of the text
-        extracted dumps (there are currently 96 dumps) we obtained roughly 36 trillion tokens of data (when
-        tokenized with the <code>gpt2</code> tokenizer).</p>
+        extracted dumps (there are currently 96 dumps) we obtained roughly 36 trillion tokens of data<d-footnote>As everywhere in this report: this is the number of tokens when tokenized with the <code>gpt2</code> tokenizer</d-footnote>.</p>
     <h3>Deduplication</h3>
     <p>Deduplication is one of the most important steps when creating large web datasets for LLM pretraining. Methods to deduplicate datasets attempt to identify and remove redundant/repeated data from the dataset. </p>
     <h4>Why deduplicate?</h4>
     <p>The web has many aggregators, mirrors, templated pages or
-        just otherwise repeated content spread over different domains and webpages. Often, these duplicated pages
-        can be introduced by the crawler itself, when different links point to the same page. </p>
-    <p>Removing these duplicates (deduplicating) has been linked to an improvement in model performance<d-cite bibtex-key="lee2022deduplicating"></d-cite> and a reduction in memorization of pretraining data<d-cite bibtex-key="carlini2023quantifying"></d-cite>, which might
-        allow for better generalization. Additionally, the performance uplift obtained through deduplication can also be tied to increased training
-        efficiency: by removing duplicated content, for the same number of training tokens, a model will have seen
-        more diverse data.<d-cite bibtex-key="muennighoff2023scaling"></d-cite><d-cite bibtex-key="hernandez2022scaling"></d-cite></p>
+        just otherwise repeated content spread over different domains and webpages. Sometimes, these duplicated pages
+        can even be introduced by the crawler itself, when different links point to the same page. </p>
+    <p>Removing these duplicates (deduplicating) has been correlated with improvements in model performance<d-cite bibtex-key="lee2022deduplicating"></d-cite> and a reduction in memorization of pretraining data<d-cite bibtex-key="carlini2023quantifying"></d-cite>, which might
+        allow for better generalization. Additionally, the performance uplift obtained through deduplication can be equated to an increased training
+        efficiency: by removing duplicated content, a model can reach the same performance level with less training iteration – or equivalently, for a given number of training tokens, a model will have seen more diverse data.<d-cite bibtex-key="muennighoff2023scaling"></d-cite><d-cite bibtex-key="hernandez2022scaling"></d-cite></p>
     <p>There are different ways to identify and even define
         duplicated data. Common approaches rely on hashing techniques to speed up the process, or on building
         efficient data structures to index the data (like suffix arrays). Methods can also be “fuzzy”, by using some
         similarity metric to mark documents as duplicates, or “exact” by checking for exact matches between two
-        documents (or lines, paragraphs, or whatever other granularity level being used).</p>
+        documents (or lines, paragraphs, or whatever other granularity level being used)<d-footnote>Note that here, even when we discuss "fuzzy" deduplication, we are only employing methods that operate on character/word matches, aka surface-level text. A more complex concept of deduplication is concerned with "semantic" deduplication: comparing/removing texts which are relative to the same concepts and use for instance synonyms or periphrase. We don't discuss these topics here but note that they can be important in the field of large-scale synthetic data generation for instance (see our <a href="https://huggingface.co/blog/cosmopedia">Cosmopedia release</a> on this topic)</d-footnote>.</p>
+        
     <h4>Our deduplication parameters</h4>
-    <p>Similarly to RefinedWeb<d-cite bibtex-key="penedo2023refinedweb"></d-cite>, we decided to apply MinHash, a
-        fuzzy hash based deduplication technique that scales well and allows us to tune similarity thresholds (by changing the number and size of buckets) and the granularity of the matches (by changing the n-gram size). We chose to compute minhashes on each document’s 5-grams, using
+    <p>Following RefinedWeb<d-cite bibtex-key="penedo2023refinedweb"></d-cite>, we decided to apply MinHash, a
+        fuzzy hash based deduplication technique that scales efficiently to many CPU-nodes and allows us to tune similarity thresholds (by controlling the number and size of buckets) as well as the length of the sequences considered (by controlling the n-gram size). We chose to work on 5-grams<d-footnote>Our units are "words", computed in the <a href="https://github.com/huggingface/datatrove/blob/e9963f69f1fbab1a61339bd1b497f6e138b9f47f/src/datatrove/pipeline/dedup/minhash.py#L196">MinHash processing function</a> with a <a href="https://github.com/huggingface/datatrove/blob/e9963f69f1fbab1a61339bd1b497f6e138b9f47f/src/datatrove/utils/word_tokenizers.py#L323">language-specific word tokenizer</a>.</d-footnote> and compute minhashes using
         112 hash functions in total, split into 14 buckets of 8 hashes each — targeting documents that are at least
         75% similar. Documents with the same 8 minhashes in any bucket are considered a duplicate of each other.</p>
     <p>This would mean that for two documents with a similarity ($$s$$)
@@ -278,28 +278,24 @@
         allows for a steeper, more well defined cut off (documents with real similarity near the threshold are more likely to be correctly identified), we believe the compute and storage savings are a reasonable
         trade off.</p>
     <p>It should also be noted that intra-document deduplication is already handled by our repetition filter, which removes documents with many repeated lines and paragraphs.</p>
+    
     <h4>More deduplication is always better, right?</h4>
-    <p>Our initial approach was to take the entire dataset (all
+    <p>We started the project with the assumption that <em>more deduplication is always better</em>, so our initial approach was to take the entire dataset (all
         90+ dumps) and deduplicate them together as one big dataset using MinHash.</p>
     <p>We did this in an iterative manner: starting with the most
-        recent dump (which at the time was 2023-50) and proceeding chronologically until the oldest one, we would deduplicate each dump
-        not only within itself, but we would also remove any matches with documents from the previously processed (more recent)
+        recent dump (which at the time was 2023-50) and proceeding chronologically until we reached the oldest crawl. We deduplicated each dump
+        not only within itself, but removing any document matching any other documents in the previously processed
         dumps. </p>
     <p>For instance, for the second most recent dump (2023-40 at
-        the time), we deduplicated it against the most recent one in addition to within itself. In particular, the oldest
-        dump was deduplicated against all other dumps. As a result, more data was removed from the oldest dumps (last
-        to be deduplicated) than from the most recent ones.</p>
+        the time), we deduplicated it against the most recent one in addition to within itself. As a result, the older the dumps, the higher the number of dumps it was deduplicated against and the more we removed data from it (indeed, in the oldest dumps we removed more than 90% of the data in the deduplication step).</p>
     <p>Deduplicating the dataset in this manner resulted in 4
-        trillion tokens of data, but, quite surprisingly for us, when training on a randomly sampled 350 billion
-        tokens subset, the model showed no improvement over one trained on the non deduplicated data (see orange and
-        green curves below), scoring far below its predecessor RefinedWeb on our aggregate of tasks.</p>
+        trillion tokens of data, but, quite surprisingly to us, when training on a randomly sampled 350 billion
+        tokens subset, our ablation models showed no improvement over a model trained on the non deduplicated data, scoring far below its predecessor RefinedWeb on our aggregate of tasks (see graph below).</p>
     <div class="main-plot-container">
         <figure><img src="assets/images/dedup_all_dumps_bad.png"/></figure>
         <div id="plot-all_dumps_bad"></div>
     </div>
-    <p>This was quite puzzling as our intuition regarding web
-        data was that more deduplication would always result in improved performance. We decided to take a closer
-        look at one of the oldest dumps, dump 2013-48:</p>
+    <p>This was challenging our assumption that more deduplication was always better so we decided to take a closer look at one of the oldest dumps, dump 2013-48:</p>
     <ul>
         <li>pre deduplication, this dump had ~490 billion tokens</li>
     </ul>
@@ -326,23 +322,22 @@
         <figure><img src="assets/images/removed_data_cross_dedup.png"/></figure>
         <div id="plot-removed_data_dedup"></div>
     </div>
-    <p>These results show that, for this older dump from which we had
-        removed over 90% of the original data, the data that was kept was actually <em>worse</em> than the data
-        removed (considered independently of all the other dumps). This is also confirmed by visual inspection: <em>originally kept
+    <p>These results show that, for this older dump taken in isolation, the data that was kept (10% of the original data) was actually <em>worse</em> than the 90% of data we
+        removed<d-footnote>Note that these ablation models are trained only on data from this dump so it's considered independently of all the other dumps.</d-footnote>. This is also confirmed by visual inspection: <em>originally kept
             data</em> contains far more ads, lists of keywords and generally badly formatted text than <em>originally removed data</em>.</p>
     <h4>Taking a step back: individual dump dedup</h4>
-    <p>We then tried an alternative approach: we deduplicated
-        each dump with MinHash individually (without considering the other dumps). This resulted in 20 trillion
+    <p>We decided to experimence with alternative approaches: we deduplicated
+        each dump with MinHash individually (independently of the other dumps). This resulted in 20 trillion
         tokens of data.</p>
     <p>When training on a random sample from this dataset we see
-        that it now matches RefinedWeb’s performance (blue and red curves below):</p>
+        that it now matches RefinedWeb’s performance (see curves below):</p>
     <div class="main-plot-container">
         <figure><img src="assets/images/cross_ind_unfiltered_comparison.png"/></figure>
         <div id="plot-ind_dedup_better"></div>
     </div>
     <p>We hypothesize that the main improvement gained from
         deduplication is the removal of very large clusters that are present in every single dump (you will find
-        some examples of these clusters on the RefinedWeb paper, each containing <em>hundreds of thousands</em> of
+        some examples of these clusters in the RefinedWeb paper, each containing <em>hundreds of thousands</em> of
         documents) and that further deduplication for clusters with a low number of duplicates (less than ~100 i.e. the number
         of dumps) actually harms performance: data that does not find a duplicate match in any other dump might
         actually be worse quality/more out of distribution (as evidenced by the results on the 2013-48 data). </p>
@@ -353,7 +348,8 @@
         improves, this effect may not be as prevalent, since the filtering might be able to remove some of this
         lower quality data. We also experimented with applying different, and often “lighter”, deduplication
         approaches on top of the individually deduplicated dumps. You can read about them further below.</p>
-    <h4>A note on measuring the effect of deduplication</h4>
+    
+        <h4>A note on measuring the effect of deduplication</h4>
     <p>Given the nature of deduplication, its effect is not
         always very visible in a smaller slice of the dataset (such as 28B tokens, the size we used for our
         filtering ablations). Furthermore, one must consider the fact that there are specific effects at play when
@@ -366,7 +362,7 @@
     </ul>
     <ul>
         <li>each dump has been perfectly individually deduplicated (every single
-            document in it is unique)
+            document in a is unique in this dump)
         </li>
     </ul>
     <ul>
@@ -401,9 +397,10 @@
         documents duplicated up to 8 times. This simulation illustrates the inherent difficulties associated with
         measuring deduplication impact on the training of LLMs, once the biggest duplicate clusters have been
         removed.</p>
+
     <h4>Other (failed) global approaches</h4>
-    <p>We attempted to improve the performance of the
-        independently minhash deduped 20 trillion tokens of data by further deduplicating it (globally, over all dumps) with the following methods:</p>
+    <p>To build on top of our newly found method (independently deduplicating each dump). We attempted to further improve the performance further deduplicating the
+        independently minhash deduped 20 trillion tokens of data (globally, over all dumps). We explored the following methods:</p>
     <ul>
         <li>URL deduplication, where we only kept one document per normalized
             (lowercased) URL (71.5% of tokens removed, 5.6T left) — <em>FineWeb URL dedup</em></li>
@@ -433,10 +430,10 @@
         <figure><img src="assets/images/dedup_attempts.png"/></figure>
         <div id="plot-dedup_attempts"></div>
     </div>
+
     <h3>Additional filtering</h3>
-    <p>By this point we had reached the same performance as
-        RefinedWeb with base filtering + independent MinHash, but on our aggregate of tasks, another heavily filtered dataset, the C4 dataset<d-cite bibtex-key="raffel2023exploring"></d-cite>, still showed stronger performance (with
-        the caveat that it is a relatively small dataset for current web-scale standards).</p>
+    <p>By this point we had reached the same performance of the previous work we attempted to reproduce and extend:
+        RefinedWeb, using our base filtering and independent MinHash. Still, on our aggregate of tasks, another heavily filtered dataset, the C4 dataset<d-cite bibtex-key="raffel2023exploring"></d-cite>, still showed stronger performances on some benchmarks of our evaluation suite.</p>
     <p>We therefore set out to find new filtering steps that
         would, at first, allow us to match the performance of C4 and, at a second stage, surpass it. A natural starting point
         was to look into the processing of C4 itself.</p>
@@ -457,8 +454,7 @@
     <ul>
         <li>applying “All filters” (drop lines not ending on punctuation marks,
             mentioning javascript and cookie notices + drop documents outside length thresholds, containing “lorem
-            ipsum” or a curly bracket, <code>{</code>) allows us to match C4’s HellaSwag performance (purple versus
-            pink curves).
+            ipsum” or a curly bracket, <code>{</code>) allows us to match C4’s HellaSwag performance ("All filter" versus "C4" curves).
         </li>
     </ul>
     <ul>
@@ -486,14 +482,13 @@
         the next section.</p>
     <h4>A statistical approach to develop heuristic filters</h4>
     <p>To develop new heuristic filters and select their thresholds we devised a systematic process:</p>
-    <ol><li>we started by collecting a very large list of high level statistics (over <strong>50</strong>) ranging from common document-level
-        metrics (e.g. number of lines, avg. line/word length, etc) to inter-document repetition metrics (MassiveText
-        inspired), on both a high quality and a lower quality web dataset;</li>
+    <ol><li>we started by collecting a very large list of high level statistics of our datasets (over <strong>fifty</strong> different metrics) ranging from common document-level
+        metrics (e.g. number of lines, avg. line/word length, etc) to inter-document repetition metrics (inspired by MassiveText), on both a high quality and a lower quality web dataset;</li>
         <li>we selected the metrics for which the Wasserstein distance between the two distributions (of the metric computed on each dataset) was larger;</li>
         <li>we inspected the histograms of the two distributions and empirically chose a threshold that would make the lower quality dataset more closely resemble the higher quality one on this metric;</li>
         <li>we validated the resulting filter (metric-threshold pair) by using it on a reference dataset and running small ablations.</li>
     </ol>
-    <p>Due to our assumption that global MinHash greatly upsamples lower quality data in the oldest dumps, we computed metrics on both the independently
+    <p>Due to our (new) assumption that global MinHash greatly upsamples lower quality data in the oldest dumps, we computed metrics on both the independently
         MinHashed and the (worse quality) global MinHashed versions of the 2013-48 and 2015-22 crawls (two older crawls). We then compared the
         statistics at a macro level, by looking at the distribution of these metrics for each one.</p>
     <p>Perhaps not too surprisingly given our findings for deduplication, we found significant
@@ -611,7 +606,7 @@
         <img src="https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/fjZQ4izIj1rx1xQnBTKKr.png" alt="Prompt for LLM annotation" style="width: 90%; max-width: 800px; height: auto;">
         <figcaption style="font-style: italic; margin-top: 10px;">Prompt used for Llama3 annotations of the educational score, also available <a href="https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/blob/main/utils/prompt.txt">here</a>.</figcaption>
     </div>    
-    <p>We also experimented with  <a href="https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1">Mixtral-8x-7B-Instruct</a> and <a href="https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1">Mixtral-8x22B-Instruct</a> and a jury of all three models<d-cite bibtex-key="verga2024replacing"></d-cite> but found that Llama3 alone gave the most reliable results.</p>
+    <p>We also experimented with  <a href="https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1">Mixtral-8x7B-Instruct</a> and <a href="https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1">Mixtral-8x22B-Instruct</a> and a jury of all three models<d-cite bibtex-key="verga2024replacing"></d-cite> but found that Llama3 alone gave the most reliable results.</p>
     <h3>Classifier Training</h3>
     <p>We added a classification head with a single regression output to <a href="https://huggingface.co/Snowflake/snowflake-arctic-embed-m">Snowflake-arctic-embed</a> and trained it on 450,000 Llama 3 annotations for 20 epochs with a learning rate of 3e-4, freezing the embedding and encoder layers.  We saved the checkpoint with the highest F1 score on our held-out validation set of 45k samples, treating Llama 3 annotations as ground-truth. After training, we rounded the scores to integers from 0 to 5.</p>
     <p>We then converted the problem to a binary classification task by using a fixed threshold to determine if a file is educational. With a threshold of 3, the model achieved an F1 score of 82% on the validation set, indicating strong performance in distinguishing high-quality educational content.</p>
@@ -624,7 +619,8 @@
         </figure>
         <div id="plot-edu-8k"></div>
     </div>
-    <p>We then built  📚 FineWeb-Edu by filtering out samples with scores lower than 3. This removed 92% of the dataset, leaving us with 1.3 trillion educational tokens.  To evaluate the effectiveness of this filtering at a larger scale, we conducted an ablation using a 1.82B model trained on 350 billion tokens, similar to the FineWeb filtering ablation mentioned above:</p>
+    <p><strong>Note:</strong> this ablation was conducted on 8B tokens from the 2024-10 dump for both the FineWeb and FineWeb-Edu subsets, which might not be representative of the entire dataset. The next ablation shows that the findings for threshold 3 hold on a longer run of 350B tokens from all FineWeb dumps, except for HellaSwag, where we noticed a slight performance degradation.</p>
+    <p>We built  📚 FineWeb-Edu by filtering out samples with scores lower than 3. This removed 92% of the dataset, leaving us with 1.3 trillion educational tokens.  To evaluate the effectiveness of this filtering at a larger scale, we conducted an ablation using a 1.82B model trained on 350 billion tokens, similar to the FineWeb filtering ablation mentioned above:</p>
     <div class="main-plot-container">
         <figure>
             <img src="assets/images/edu-100k.png">

src/index.html

@@ -108,7 +108,7 @@
         releases a new crawl containing 200 to 400 TiB of textual content obtained via automatic web crawling usually
         every 1 or 2 months. </p>
     <p>As an example, the latest CC crawl (April 2024) contains 2.7
-        billion web pages, totaling 386 TiB of uncompressed HTML text content<d-footnote>Note that the size changes from crawl to crawl</d-footnote>.
+        billion web pages, totaling 386 TiB of uncompressed HTML text content<d-footnote>Note that the size changes from crawl to crawl. Note also that we use "dump" or "crawl" interchangeability in this report.</d-footnote>.
       Ninety-six crawls have been released since 2013 and 3 crawls from 2008 to 2012, which are in a different (older) format.
       <d-footnote>We have not processed these 3 older crawls.</d-footnote> </p>
   
@@ -150,7 +150,7 @@
         scores.</p>
     <p>Our ablation models were trained using <a
             href="https://github.com/huggingface/nanotron"><code>nanotron</code></a> with this config [<strong>TODO:
-        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Our ablation models have 1.82B parameters (including embeddings), used the Llama
+        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Our "ablation models" have 1.82B parameters (including embeddings), used the Llama
         architecture with a 2048 sequence length, a global batch size of ~2 million tokens, and the GPT2 tokenizer. For most
         ablations we trained on ~28B tokens (roughly the Chinchilla<d-cite bibtex-key="hoffmann2022training"></d-cite> optimal training size for this
         model size). To confirm relative performance improvements after each step of filtering we conducted longer training runs on 350 billion tokens as mentioned further below.</p>
@@ -161,7 +161,7 @@
     <ul>
         <li>small variance between runs trained on different samplings of the same
             dataset: we want our runs on a subset of the data to be representative of the whole dataset, and the
-            resulting scores to have as little sensitivity to exact data choice as possible (apart from larger ablations that we are concerned with)
+            resulting scores to be, in the limit of what is possible, less sensitive to exact data point choices than to our filter ablations.
         </li>
     </ul>
     <ul>
@@ -204,10 +204,10 @@
         full page HTML and request metadata. <strong>WET</strong> (WARC Encapsulated Text) files provide a text only
         version of those websites.</p>
     <p>A large number of datasets take the WET files as their
-        starting point. In our experience the default text extraction (extracting the main text of a webpage from
-        its HTML) used to create these WET files is suboptimal and there are a variety of open-source libraries that
-        provide better text extraction (by, namely, keeping less boilerplate content/navigation menus). We extracted
-        the text content from the WARC files using the trafilatura library<d-cite bibtex-key="barbaresi-2021-trafilatura"></d-cite>, which from visual inspection of the results provided good quality extraction when compared to other libraries.</p><aside>You can also find a benchmark on text extraction libraries <a href="https://github.com/scrapinghub/article-extraction-benchmark/blob/master/README.rst">here</a>.</aside>
+        starting point. In our experience the default text extraction used by Common Crawl to create these WET files is suboptimal for the goals of LLM pretraining<d-footnote>In particular we suspect that it keeps too much boilerplate content and navigation menus.</d-footnote> and there are a variety of open-source libraries that
+        provide better text extraction. We extracted
+        the text content from the WARC files using the trafilatura library<d-cite bibtex-key="barbaresi-2021-trafilatura"></d-cite>, which from visual inspection of the results provided good quality extraction when compared to other libraries.</p>
+    <aside>You can find a benchmark comparing several text extraction libraries <a href="https://github.com/scrapinghub/article-extraction-benchmark/blob/master/README.rst">here</a>.</aside>
     <p>To validate this decision, we processed the 2019-18 dump
         directly using the WET files and with text extracted from WARC files using trafilatura<d-footnote>We used trafilatura default options with <code>favour_precision=True</code>.</d-footnote>. We applied the same
         processing to each one (our base filtering+minhash, detailed below) and trained two models. While the
@@ -221,10 +221,11 @@
         <figure><img src="assets/images/wet_comparison.png"/></figure>
         <div id="plot-wet_comparison"></div>
     </div>
+
     <h3>Base filtering</h3>
-    <p>Filtering is an important part of the curation process. It
-        removes part of the data (be it words, lines, or full documents) that would harm performance and is thus
-        deemed to be “lower quality”.</p>
+    <p>Filtering is an important part of the curation process. It consists in
+        removing part of the data (which can consists in removing words, lines, or even full documents) that lower the performances of the model and is thus
+        deemed to be “lower quality” in our eval-driven process of dataset crafting.</p>
     <p>As a basis for our filtering we used part of the setup
         from RefinedWeb<d-cite bibtex-key="penedo2023refinedweb"></d-cite>. Namely, we:</p>
     <ul>
@@ -243,26 +244,25 @@
         </li>
     </ul>
     <p>After applying this filtering to each of the text
-        extracted dumps (there are currently 96 dumps) we obtained roughly 36 trillion tokens of data (when
-        tokenized with the <code>gpt2</code> tokenizer).</p>
+        extracted dumps (there are currently 96 dumps) we obtained roughly 36 trillion tokens of data<d-footnote>As everywhere in this report: this is the number of tokens when tokenized with the <code>gpt2</code> tokenizer</d-footnote>.</p>
     <h3>Deduplication</h3>
     <p>Deduplication is one of the most important steps when creating large web datasets for LLM pretraining. Methods to deduplicate datasets attempt to identify and remove redundant/repeated data from the dataset. </p>
     <h4>Why deduplicate?</h4>
     <p>The web has many aggregators, mirrors, templated pages or
-        just otherwise repeated content spread over different domains and webpages. Often, these duplicated pages
-        can be introduced by the crawler itself, when different links point to the same page. </p>
-    <p>Removing these duplicates (deduplicating) has been linked to an improvement in model performance<d-cite bibtex-key="lee2022deduplicating"></d-cite> and a reduction in memorization of pretraining data<d-cite bibtex-key="carlini2023quantifying"></d-cite>, which might
-        allow for better generalization. Additionally, the performance uplift obtained through deduplication can also be tied to increased training
-        efficiency: by removing duplicated content, for the same number of training tokens, a model will have seen
-        more diverse data.<d-cite bibtex-key="muennighoff2023scaling"></d-cite><d-cite bibtex-key="hernandez2022scaling"></d-cite></p>
+        just otherwise repeated content spread over different domains and webpages. Sometimes, these duplicated pages
+        can even be introduced by the crawler itself, when different links point to the same page. </p>
+    <p>Removing these duplicates (deduplicating) has been correlated with improvements in model performance<d-cite bibtex-key="lee2022deduplicating"></d-cite> and a reduction in memorization of pretraining data<d-cite bibtex-key="carlini2023quantifying"></d-cite>, which might
+        allow for better generalization. Additionally, the performance uplift obtained through deduplication can be equated to an increased training
+        efficiency: by removing duplicated content, a model can reach the same performance level with less training iteration – or equivalently, for a given number of training tokens, a model will have seen more diverse data.<d-cite bibtex-key="muennighoff2023scaling"></d-cite><d-cite bibtex-key="hernandez2022scaling"></d-cite></p>
     <p>There are different ways to identify and even define
         duplicated data. Common approaches rely on hashing techniques to speed up the process, or on building
         efficient data structures to index the data (like suffix arrays). Methods can also be “fuzzy”, by using some
         similarity metric to mark documents as duplicates, or “exact” by checking for exact matches between two
-        documents (or lines, paragraphs, or whatever other granularity level being used).</p>
+        documents (or lines, paragraphs, or whatever other granularity level being used)<d-footnote>Note that here, even when we discuss "fuzzy" deduplication, we are only employing methods that operate on character/word matches, aka surface-level text. A more complex concept of deduplication is concerned with "semantic" deduplication: comparing/removing texts which are relative to the same concepts and use for instance synonyms or periphrase. We don't discuss these topics here but note that they can be important in the field of large-scale synthetic data generation for instance (see our <a href="https://huggingface.co/blog/cosmopedia">Cosmopedia release</a> on this topic)</d-footnote>.</p>
+        
     <h4>Our deduplication parameters</h4>
-    <p>Similarly to RefinedWeb<d-cite bibtex-key="penedo2023refinedweb"></d-cite>, we decided to apply MinHash, a
-        fuzzy hash based deduplication technique that scales well and allows us to tune similarity thresholds (by changing the number and size of buckets) and the granularity of the matches (by changing the n-gram size). We chose to compute minhashes on each document’s 5-grams, using
+    <p>Following RefinedWeb<d-cite bibtex-key="penedo2023refinedweb"></d-cite>, we decided to apply MinHash, a
+        fuzzy hash based deduplication technique that scales efficiently to many CPU-nodes and allows us to tune similarity thresholds (by controlling the number and size of buckets) as well as the length of the sequences considered (by controlling the n-gram size). We chose to work on 5-grams<d-footnote>Our units are "words", computed in the <a href="https://github.com/huggingface/datatrove/blob/e9963f69f1fbab1a61339bd1b497f6e138b9f47f/src/datatrove/pipeline/dedup/minhash.py#L196">MinHash processing function</a> with a <a href="https://github.com/huggingface/datatrove/blob/e9963f69f1fbab1a61339bd1b497f6e138b9f47f/src/datatrove/utils/word_tokenizers.py#L323">language-specific word tokenizer</a>.</d-footnote> and compute minhashes using
         112 hash functions in total, split into 14 buckets of 8 hashes each — targeting documents that are at least
         75% similar. Documents with the same 8 minhashes in any bucket are considered a duplicate of each other.</p>
     <p>This would mean that for two documents with a similarity ($$s$$)
@@ -278,28 +278,24 @@
         allows for a steeper, more well defined cut off (documents with real similarity near the threshold are more likely to be correctly identified), we believe the compute and storage savings are a reasonable
         trade off.</p>
     <p>It should also be noted that intra-document deduplication is already handled by our repetition filter, which removes documents with many repeated lines and paragraphs.</p>
+    
     <h4>More deduplication is always better, right?</h4>
-    <p>Our initial approach was to take the entire dataset (all
+    <p>We started the project with the assumption that <em>more deduplication is always better</em>, so our initial approach was to take the entire dataset (all
         90+ dumps) and deduplicate them together as one big dataset using MinHash.</p>
     <p>We did this in an iterative manner: starting with the most
-        recent dump (which at the time was 2023-50) and proceeding chronologically until the oldest one, we would deduplicate each dump
-        not only within itself, but we would also remove any matches with documents from the previously processed (more recent)
+        recent dump (which at the time was 2023-50) and proceeding chronologically until we reached the oldest crawl. We deduplicated each dump
+        not only within itself, but removing any document matching any other documents in the previously processed
         dumps. </p>
     <p>For instance, for the second most recent dump (2023-40 at
-        the time), we deduplicated it against the most recent one in addition to within itself. In particular, the oldest
-        dump was deduplicated against all other dumps. As a result, more data was removed from the oldest dumps (last
-        to be deduplicated) than from the most recent ones.</p>
+        the time), we deduplicated it against the most recent one in addition to within itself. As a result, the older the dumps, the higher the number of dumps it was deduplicated against and the more we removed data from it (indeed, in the oldest dumps we removed more than 90% of the data in the deduplication step).</p>
     <p>Deduplicating the dataset in this manner resulted in 4
-        trillion tokens of data, but, quite surprisingly for us, when training on a randomly sampled 350 billion
-        tokens subset, the model showed no improvement over one trained on the non deduplicated data (see orange and
-        green curves below), scoring far below its predecessor RefinedWeb on our aggregate of tasks.</p>
+        trillion tokens of data, but, quite surprisingly to us, when training on a randomly sampled 350 billion
+        tokens subset, our ablation models showed no improvement over a model trained on the non deduplicated data, scoring far below its predecessor RefinedWeb on our aggregate of tasks (see graph below).</p>
     <div class="main-plot-container">
         <figure><img src="assets/images/dedup_all_dumps_bad.png"/></figure>
         <div id="plot-all_dumps_bad"></div>
     </div>
-    <p>This was quite puzzling as our intuition regarding web
-        data was that more deduplication would always result in improved performance. We decided to take a closer
-        look at one of the oldest dumps, dump 2013-48:</p>
+    <p>This was challenging our assumption that more deduplication was always better so we decided to take a closer look at one of the oldest dumps, dump 2013-48:</p>
     <ul>
         <li>pre deduplication, this dump had ~490 billion tokens</li>
     </ul>
@@ -326,23 +322,22 @@
         <figure><img src="assets/images/removed_data_cross_dedup.png"/></figure>
         <div id="plot-removed_data_dedup"></div>
     </div>
-    <p>These results show that, for this older dump from which we had
-        removed over 90% of the original data, the data that was kept was actually <em>worse</em> than the data
-        removed (considered independently of all the other dumps). This is also confirmed by visual inspection: <em>originally kept
+    <p>These results show that, for this older dump taken in isolation, the data that was kept (10% of the original data) was actually <em>worse</em> than the 90% of data we
+        removed<d-footnote>Note that these ablation models are trained only on data from this dump so it's considered independently of all the other dumps.</d-footnote>. This is also confirmed by visual inspection: <em>originally kept
             data</em> contains far more ads, lists of keywords and generally badly formatted text than <em>originally removed data</em>.</p>
     <h4>Taking a step back: individual dump dedup</h4>
-    <p>We then tried an alternative approach: we deduplicated
-        each dump with MinHash individually (without considering the other dumps). This resulted in 20 trillion
+    <p>We decided to experimence with alternative approaches: we deduplicated
+        each dump with MinHash individually (independently of the other dumps). This resulted in 20 trillion
         tokens of data.</p>
     <p>When training on a random sample from this dataset we see
-        that it now matches RefinedWeb’s performance (blue and red curves below):</p>
+        that it now matches RefinedWeb’s performance (see curves below):</p>
     <div class="main-plot-container">
         <figure><img src="assets/images/cross_ind_unfiltered_comparison.png"/></figure>
         <div id="plot-ind_dedup_better"></div>
     </div>
     <p>We hypothesize that the main improvement gained from
         deduplication is the removal of very large clusters that are present in every single dump (you will find
-        some examples of these clusters on the RefinedWeb paper, each containing <em>hundreds of thousands</em> of
+        some examples of these clusters in the RefinedWeb paper, each containing <em>hundreds of thousands</em> of
         documents) and that further deduplication for clusters with a low number of duplicates (less than ~100 i.e. the number
         of dumps) actually harms performance: data that does not find a duplicate match in any other dump might
         actually be worse quality/more out of distribution (as evidenced by the results on the 2013-48 data). </p>
@@ -353,7 +348,8 @@
         improves, this effect may not be as prevalent, since the filtering might be able to remove some of this
         lower quality data. We also experimented with applying different, and often “lighter”, deduplication
         approaches on top of the individually deduplicated dumps. You can read about them further below.</p>
-    <h4>A note on measuring the effect of deduplication</h4>
+    
+        <h4>A note on measuring the effect of deduplication</h4>
     <p>Given the nature of deduplication, its effect is not
         always very visible in a smaller slice of the dataset (such as 28B tokens, the size we used for our
         filtering ablations). Furthermore, one must consider the fact that there are specific effects at play when
@@ -366,7 +362,7 @@
     </ul>
     <ul>
         <li>each dump has been perfectly individually deduplicated (every single
-            document in it is unique)
+            document in a is unique in this dump)
         </li>
     </ul>
     <ul>
@@ -401,9 +397,10 @@
         documents duplicated up to 8 times. This simulation illustrates the inherent difficulties associated with
         measuring deduplication impact on the training of LLMs, once the biggest duplicate clusters have been
         removed.</p>
+
     <h4>Other (failed) global approaches</h4>
-    <p>We attempted to improve the performance of the
-        independently minhash deduped 20 trillion tokens of data by further deduplicating it (globally, over all dumps) with the following methods:</p>
+    <p>To build on top of our newly found method (independently deduplicating each dump). We attempted to further improve the performance further deduplicating the
+        independently minhash deduped 20 trillion tokens of data (globally, over all dumps). We explored the following methods:</p>
     <ul>
         <li>URL deduplication, where we only kept one document per normalized
             (lowercased) URL (71.5% of tokens removed, 5.6T left) — <em>FineWeb URL dedup</em></li>
@@ -433,10 +430,10 @@
         <figure><img src="assets/images/dedup_attempts.png"/></figure>
         <div id="plot-dedup_attempts"></div>
     </div>
+
     <h3>Additional filtering</h3>
-    <p>By this point we had reached the same performance as
-        RefinedWeb with base filtering + independent MinHash, but on our aggregate of tasks, another heavily filtered dataset, the C4 dataset<d-cite bibtex-key="raffel2023exploring"></d-cite>, still showed stronger performance (with
-        the caveat that it is a relatively small dataset for current web-scale standards).</p>
+    <p>By this point we had reached the same performance of the previous work we attempted to reproduce and extend:
+        RefinedWeb, using our base filtering and independent MinHash. Still, on our aggregate of tasks, another heavily filtered dataset, the C4 dataset<d-cite bibtex-key="raffel2023exploring"></d-cite>, still showed stronger performances on some benchmarks of our evaluation suite.</p>
     <p>We therefore set out to find new filtering steps that
         would, at first, allow us to match the performance of C4 and, at a second stage, surpass it. A natural starting point
         was to look into the processing of C4 itself.</p>
@@ -457,8 +454,7 @@
     <ul>
         <li>applying “All filters” (drop lines not ending on punctuation marks,
             mentioning javascript and cookie notices + drop documents outside length thresholds, containing “lorem
-            ipsum” or a curly bracket, <code>{</code>) allows us to match C4’s HellaSwag performance (purple versus
-            pink curves).
+            ipsum” or a curly bracket, <code>{</code>) allows us to match C4’s HellaSwag performance ("All filter" versus "C4" curves).
         </li>
     </ul>
     <ul>
@@ -486,14 +482,13 @@
         the next section.</p>
     <h4>A statistical approach to develop heuristic filters</h4>
     <p>To develop new heuristic filters and select their thresholds we devised a systematic process:</p>
-    <ol><li>we started by collecting a very large list of high level statistics (over <strong>50</strong>) ranging from common document-level
-        metrics (e.g. number of lines, avg. line/word length, etc) to inter-document repetition metrics (MassiveText
-        inspired), on both a high quality and a lower quality web dataset;</li>
+    <ol><li>we started by collecting a very large list of high level statistics of our datasets (over <strong>fifty</strong> different metrics) ranging from common document-level
+        metrics (e.g. number of lines, avg. line/word length, etc) to inter-document repetition metrics (inspired by MassiveText), on both a high quality and a lower quality web dataset;</li>
         <li>we selected the metrics for which the Wasserstein distance between the two distributions (of the metric computed on each dataset) was larger;</li>
         <li>we inspected the histograms of the two distributions and empirically chose a threshold that would make the lower quality dataset more closely resemble the higher quality one on this metric;</li>
         <li>we validated the resulting filter (metric-threshold pair) by using it on a reference dataset and running small ablations.</li>
     </ol>
-    <p>Due to our assumption that global MinHash greatly upsamples lower quality data in the oldest dumps, we computed metrics on both the independently
+    <p>Due to our (new) assumption that global MinHash greatly upsamples lower quality data in the oldest dumps, we computed metrics on both the independently
         MinHashed and the (worse quality) global MinHashed versions of the 2013-48 and 2015-22 crawls (two older crawls). We then compared the
         statistics at a macro level, by looking at the distribution of these metrics for each one.</p>
     <p>Perhaps not too surprisingly given our findings for deduplication, we found significant