January 2025 data update

README.md

@@ -8,14 +8,14 @@ library_name: txtai
 tags:
 - sentence-similarity
 datasets:
-- NeuML/wikipedia-20240901
+- NeuML/wikipedia-20250123
 ---
 
 # Wikipedia txtai embeddings index
 
 This is a [txtai](https://github.com/neuml/txtai) embeddings index for the [English edition of Wikipedia](https://en.wikipedia.org/).
 
-This index is built from the [Wikipedia September 2024 dataset](https://huggingface.co/datasets/neuml/wikipedia-20240901). Only the first paragraph of the [lead section](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Lead_section) from each article is included in the index. This is similar to an abstract of the article.
+This index is built from the [Wikipedia January 2025 dataset](https://huggingface.co/datasets/neuml/wikipedia-20250123). Only the first paragraph of the [lead section](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Lead_section) from each article is included in the index. This is similar to an abstract of the article.
 
 It also uses [Wikipedia Page Views](https://dumps.wikimedia.org/other/pageviews/readme.html) data to add a `percentile` field. The `percentile` field can be used
 to only match commonly visited pages.
@@ -65,7 +65,7 @@ Performance was evaluated using the [NDCG@10](https://en.wikipedia.org/wiki/Disc
 
 ## Build the index
 
-The following steps show how to build this index. These scripts are using the latest data available as of 2024-09-01, update as appropriate.
+The following steps show how to build this index. These scripts are using the latest data available as of 2025-01-23, update as appropriate.
 
 - Install required build dependencies
 ```bash
@@ -75,7 +75,7 @@ pip install ragdata mwparserfromhell
 - Download and build pageviews database
 ```bash
 mkdir -p pageviews/data
-wget -P pageviews/data https://dumps.wikimedia.org/other/pageview_complete/monthly/2024/2024-08/pageviews-202408-user.bz2
+wget -P pageviews/data https://dumps.wikimedia.org/other/pageview_complete/monthly/2025/2025-01/pageviews-202501-user.bz2
 python -m ragdata.wikipedia.views -p en.wikipedia -v pageviews
 ```
 
@@ -85,7 +85,7 @@ python -m ragdata.wikipedia.views -p en.wikipedia -v pageviews
 from datasets import load_dataset
 
 # Data dump date from https://dumps.wikimedia.org/enwiki/
-date = "20240901"
+date = "20250123"
 
 # Build and save dataset
 ds = load_dataset("neuml/wikipedia", language="en", date=date)
@@ -95,7 +95,7 @@ ds.save_to_disk(f"wikipedia-{date}")
 - Build txtai-wikipedia index
 ```bash
 python -m ragdata.wikipedia.index \
-       -d wikipedia-20240901 \
+       -d wikipedia-20250123 \
        -o txtai-wikipedia \
        -v pageviews/pageviews.sqlite
 ```

config.json

@@ -14,15 +14,15 @@
   "content": true,
   "dimensions": 768,
   "backend": "faiss",
-  "offset": 6272285,
+  "offset": 6333058,
   "build": {
-    "create": "2024-09-12T17:06:38Z",
-    "python": "3.8.19",
+    "create": "2025-02-01T17:41:48Z",
+    "python": "3.9.21",
     "settings": {
-      "components": "IVF2240,SQ8"
+      "components": "IVF2251,SQ8"
     },
     "system": "Linux (x86_64)",
-    "txtai": "7.4.0"
+    "txtai": "8.2.0"
   },
-  "update": "2024-09-12T17:06:38Z"
+  "update": "2025-02-01T17:41:48Z"
 }

documents

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a54c4473c76d15a7bf2d1b8a1b590d3aaeacc0426324c4a5b1d886d729a43b92
-size 3292749824
+oid sha256:3e05be4645d78cfc665f78a119528833a23b6aa2c4e0c91477988af4bdb917d5
+size 3406143488

embeddings

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:47bcc78a1602223ac3b2b8355b3c244a62fb2e4d0e2f04cdc8a8d0a865692b35
-size 4874198688
+oid sha256:fb24f6eeb967b08fdf83c7b6e698f4f2c3391d55ebbc903225428a9733b3b909
+size 4921392416