Training in progress, step 25, checkpoint

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: echarlaix/tiny-random-PhiForCausalLM
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

last-checkpoint/adapter_config.json

@@ -0,0 +1,33 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "echarlaix/tiny-random-PhiForCausalLM",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "dense",
+    "fc1",
+    "fc2",
+    "k_proj",
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cdf14adeceef0fbd05b21f9a0ef2e534ef38247a87d4e45b391749e8c24c669
+size 186962

last-checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3b152165a63d4868526b12b251f5243c7e07f14b1bfe3a0b5f8fca8b4297e26
+size 121430

last-checkpoint/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8cd406dcc1a69f352bf665054745e7b0e58cf1a05b11c53f3d235980b0212ea3
+size 14960

last-checkpoint/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf354fc1278381f18e626930186c06ebc4a0c7f2b402864a5e2123bb5e07df19
+size 15024

last-checkpoint/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8003ab1179cb0862f43f337f894a13fa4a042517cf9ca22aaca0ef2a34dbf1ea
+size 15024

last-checkpoint/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82b0cf7205a13aa26934925e69331ac92c27447d93ac004d203c25663f0e058f
+size 15024

last-checkpoint/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ea7ff16b0c30a914eb0d145e3fb06ff9027c6cd2408e766ce8a09accab89a4d
+size 1064

last-checkpoint/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

last-checkpoint/tokenizer.json

@@ -0,0 +1,4177 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|endoftext|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<|padding|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1024,
+      "content": "[PAD]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": {
+    "type": "NFC"
+  },
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<|endoftext|>": 0,
+      "<|padding|>": 1,
+      "!": 2,
+      "\"": 3,
+      "#": 4,
+      "$": 5,
+      "%": 6,
+      "&": 7,
+      "'": 8,
+      "(": 9,
+      ")": 10,
+      "*": 11,
+      "+": 12,
+      ",": 13,
+      "-": 14,
+      ".": 15,
+      "/": 16,
+      "0": 17,
+      "1": 18,
+      "2": 19,
+      "3": 20,
+      "4": 21,
+      "5": 22,
+      "6": 23,
+      "7": 24,
+      "8": 25,
+      "9": 26,
+      ":": 27,
+      ";": 28,
+      "<": 29,
+      "=": 30,
+      ">": 31,
+      "?": 32,
+      "@": 33,
+      "A": 34,
+      "B": 35,
+      "C": 36,
+      "D": 37,
+      "E": 38,
+      "F": 39,
+      "G": 40,
+      "H": 41,
+      "I": 42,
+      "J": 43,
+      "K": 44,
+      "L": 45,
+      "M": 46,
+      "N": 47,
+      "O": 48,
+      "P": 49,
+      "Q": 50,
+      "R": 51,
+      "S": 52,
+      "T": 53,
+      "U": 54,
+      "V": 55,
+      "W": 56,
+      "X": 57,
+      "Y": 58,
+      "Z": 59,
+      "[": 60,
+      "\\": 61,
+      "]": 62,
+      "^": 63,
+      "_": 64,
+      "`": 65,
+      "a": 66,
+      "b": 67,
+      "c": 68,
+      "d": 69,
+      "e": 70,
+      "f": 71,
+      "g": 72,
+      "h": 73,
+      "i": 74,
+      "j": 75,
+      "k": 76,
+      "l": 77,
+      "m": 78,
+      "n": 79,
+      "o": 80,
+      "p": 81,
+      "q": 82,
+      "r": 83,
+      "s": 84,
+      "t": 85,
+      "u": 86,
+      "v": 87,
+      "w": 88,
+      "x": 89,
+      "y": 90,
+      "z": 91,
+      "{": 92,
+      "|": 93,
+      "}": 94,
+      "~": 95,
+      "¡": 96,
+      "¢": 97,
+      "£": 98,
+      "¤": 99,
+      "¥": 100,
+      "¦": 101,
+      "§": 102,
+      "¨": 103,
+      "©": 104,
+      "ª": 105,
+      "«": 106,
+      "¬": 107,
+      "®": 108,
+      "¯": 109,
+      "°": 110,
+      "±": 111,
+      "²": 112,
+      "³": 113,
+      "´": 114,
+      "µ": 115,
+      "¶": 116,
+      "·": 117,
+      "¸": 118,
+      "¹": 119,
+      "º": 120,
+      "»": 121,
+      "¼": 122,
+      "½": 123,
+      "¾": 124,
+      "¿": 125,
+      "À": 126,
+      "Á": 127,
+      "Â": 128,
+      "Ã": 129,
+      "Ä": 130,
+      "Å": 131,
+      "Æ": 132,
+      "Ç": 133,
+      "È": 134,
+      "É": 135,
+      "Ê": 136,
+      "Ë": 137,
+      "Ì": 138,
+      "Í": 139,
+      "Î": 140,
+      "Ï": 141,
+      "Ð": 142,
+      "Ñ": 143,
+      "Ò": 144,
+      "Ó": 145,
+      "Ô": 146,
+      "Õ": 147,
+      "Ö": 148,
+      "×": 149,
+      "Ø": 150,
+      "Ù": 151,
+      "Ú": 152,
+      "Û": 153,
+      "Ü": 154,
+      "Ý": 155,
+      "Þ": 156,
+      "ß": 157,
+      "à": 158,
+      "á": 159,
+      "â": 160,
+      "ã": 161,
+      "ä": 162,
+      "å": 163,
+      "æ": 164,
+      "ç": 165,
+      "è": 166,
+      "é": 167,
+      "ê": 168,
+      "ë": 169,
+      "ì": 170,
+      "í": 171,
+      "î": 172,
+      "ï": 173,
+      "ð": 174,
+      "ñ": 175,
+      "ò": 176,
+      "ó": 177,
+      "ô": 178,
+      "õ": 179,
+      "ö": 180,
+      "÷": 181,
+      "ø": 182,
+      "ù": 183,
+      "ú": 184,
+      "û": 185,
+      "ü": 186,
+      "ý": 187,
+      "þ": 188,
+      "ÿ": 189,
+      "Ā": 190,
+      "ā": 191,
+      "Ă": 192,
+      "ă": 193,
+      "Ą": 194,
+      "ą": 195,
+      "Ć": 196,
+      "ć": 197,
+      "Ĉ": 198,
+      "ĉ": 199,
+      "Ċ": 200,
+      "ċ": 201,
+      "Č": 202,
+      "č": 203,
+      "Ď": 204,
+      "ď": 205,
+      "Đ": 206,
+      "đ": 207,
+      "Ē": 208,
+      "ē": 209,
+      "Ĕ": 210,
+      "ĕ": 211,
+      "Ė": 212,
+      "ė": 213,
+      "Ę": 214,
+      "ę": 215,
+      "Ě": 216,
+      "ě": 217,
+      "Ĝ": 218,
+      "ĝ": 219,
+      "Ğ": 220,
+      "ğ": 221,
+      "Ġ": 222,
+      "ġ": 223,
+      "Ģ": 224,
+      "ģ": 225,
+      "Ĥ": 226,
+      "ĥ": 227,
+      "Ħ": 228,
+      "ħ": 229,
+      "Ĩ": 230,
+      "ĩ": 231,
+      "Ī": 232,
+      "ī": 233,
+      "Ĭ": 234,
+      "ĭ": 235,
+      "Į": 236,
+      "į": 237,
+      "İ": 238,
+      "ı": 239,
+      "IJ": 240,
+      "ij": 241,
+      "Ĵ": 242,
+      "ĵ": 243,
+      "Ķ": 244,
+      "ķ": 245,
+      "ĸ": 246,
+      "Ĺ": 247,
+      "ĺ": 248,
+      "Ļ": 249,
+      "ļ": 250,
+      "Ľ": 251,
+      "ľ": 252,
+      "Ŀ": 253,
+      "ŀ": 254,
+      "Ł": 255,
+      "ł": 256,
+      "Ń": 257,
+      "Ġt": 258,
+      "he": 259,
+      "Ġa": 260,
+      "in": 261,
+      "Ġthe": 262,
+      "er": 263,
+      "on": 264,
+      "Ġ,": 265,
+      "re": 266,
+      "Ġs": 267,
+      "ed": 268,
+      "Ġo": 269,
+      "Ġw": 270,
+      "nd": 271,
+      "at": 272,
+      "Ġ.": 273,
+      "or": 274,
+      "it": 275,
+      "Ġc": 276,
+      "en": 277,
+      "Ġf": 278,
+      "is": 279,
+      "es": 280,
+      "ar": 281,
+      "Ġof": 282,
+      "Ġb": 283,
+      "an": 284,
+      "Ġin": 285,
+      "al": 286,
+      "ing": 287,
+      "Ġp": 288,
+      "Ġand": 289,
+      "as": 290,
+      "Ġto": 291,
+      "ro": 292,
+      "ic": 293,
+      "Ġm": 294,
+      "Ġd": 295,
+      "Ġh": 296,
+      "ion": 297,
+      "le": 298,
+      "ou": 299,
+      "ĠT": 300,
+      "Ġre": 301,
+      "Ġ=": 302,
+      "Ġ\"": 303,
+      "ĠA": 304,
+      "ĠS": 305,
+      "ent": 306,
+      "il": 307,
+      "Ġth": 308,
+      "Ġ1": 309,
+      "st": 310,
+      "ĠC": 311,
+      "el": 312,
+      "om": 313,
+      "Ġl": 314,
+      "am": 315,
+      "ĠĊ": 316,
+      "Ġe": 317,
+      "Ġn": 318,
+      "Ġ@": 319,
+      "ad": 320,
+      "ac": 321,
+      "Ġwas": 322,
+      "ĠM": 323,
+      "ur": 324,
+      "ĠThe": 325,
+      "ec": 326,
+      "Ġon": 327,
+      "ly": 328,
+      "ĠB": 329,
+      "ĠI": 330,
+      "Ġg": 331,
+      "Ġ'": 332,
+      "et": 333,
+      "ol": 334,
+      "id": 335,
+      "iv": 336,
+      "im": 337,
+      "Ġfor": 338,
+      "ir": 339,
+      "-@": 340,
+      "Ġ@-@": 341,
+      "ig": 342,
+      "ot": 343,
+      "ter": 344,
+      "Ġas": 345,
+      "ĠH": 346,
+      "us": 347,
+      "ow": 348,
+      "Ġst": 349,
+      "ut": 350,
+      "ith": 351,
+      "ay": 352,
+      "Ġ2": 353,
+      "ĠP": 354,
+      "ation": 355,
+      "ver": 356,
+      "Ġbe": 357,
+      "her": 358,
+      "Ġthat": 359,
+      "Ġwith": 360,
+      "ĠR": 361,
+      "ce": 362,
+      "th": 363,
+      "ĠD": 364,
+      "Ġis": 365,
+      "un": 366,
+      "em": 367,
+      "ĠF": 368,
+      "Ġwh": 369,
+      "ul": 370,
+      "Ġby": 371,
+      "Ġal": 372,
+      "ch": 373,
+      "Ġ)": 374,
+      "Ġ(": 375,
+      "ĠW": 376,
+      "Ġcon": 377,
+      "ra": 378,
+      "ĠG": 379,
+      "os": 380,
+      "ĠL": 381,
+      "ĠN": 382,
+      "Ġat": 383,
+      "ers": 384,
+      "ct": 385,
+      "Ġit": 386,
+      "Ġ19": 387,
+      "rom": 388,
+      "and": 389,
+      "Ġan": 390,
+      "um": 391,
+      "est": 392,
+      "ĠJ": 393,
+      "ag": 394,
+      "Ġhe": 395,
+      "00": 396,
+      "ist": 397,
+      "ain": 398,
+      "od": 399,
+      "av": 400,
+      "ri": 401,
+      "ĠE": 402,
+      "ĠO": 403,
+      "Ġfrom": 404,
+      "Ġcom": 405,
+      "Ġhis": 406,
+      "op": 407,
+      "Ġpro": 408,
+      "res": 409,
+      "ies": 410,
+      "if": 411,
+      "Ġv": 412,
+      "ort": 413,
+      "ere": 414,
+      "ill": 415,
+      "ld": 416,
+      "Ġde": 417,
+      "pp": 418,
+      "Ġsu": 419,
+      "ore": 420,
+      "ĠIn": 421,
+      "Ġr": 422,
+      "Ġse": 423,
+      "Ġwere": 424,
+      "ew": 425,
+      "ong": 426,
+      "igh": 427,
+      "ard": 428,
+      "ate": 429,
+      "all": 430,
+      "art": 431,
+      "ak": 432,
+      "ich": 433,
+      "Ġch": 434,
+      "Ġor": 435,
+      "ab": 436,
+      "ant": 437,
+      "ud": 438,
+      "oc": 439,
+      "ber": 440,
+      "Ġex": 441,
+      "gh": 442,
+      "ity": 443,
+      "ated": 444,
+      "pt": 445,
+      "ess": 446,
+      "ear": 447,
+      "ĠK": 448,
+      "Ġpl": 449,
+      "ame": 450,
+      "qu": 451,
+      "ive": 452,
+      "rou": 453,
+      "Ġare": 454,
+      "Ġâ": 455,
+      "Ġsh": 456,
+      "Ġk": 457,
+      "ack": 458,
+      "ect": 459,
+      "ĠâĢ": 460,
+      "ĠU": 461,
+      "Ġhad": 462,
+      "se": 463,
+      "Ġwhich": 464,
+      "red": 465,
+      "ov": 466,
+      "ĠSt": 467,
+      "ast": 468,
+      "Ġsp": 469,
+      "ian": 470,
+      "Ġy": 471,
+      "ment": 472,
+      "Ġle": 473,
+      "Ġnot": 474,
+      "ge": 475,
+      "ord": 476,
+      "rit": 477,
+      "ip": 478,
+      "ine": 479,
+      "ell": 480,
+      "ally": 481,
+      "our": 482,
+      "ost": 483,
+      "ight": 484,
+      "ther": 485,
+      "ap": 486,
+      "Ġu": 487,
+      "ish": 488,
+      "ĠCh": 489,
+      "oun": 490,
+      "ia": 491,
+      "Ġ3": 492,
+      "ave": 493,
+      "ary": 494,
+      "ust": 495,
+      "og": 496,
+      "Ġ200": 497,
+      "Ġun": 498,
+      "ous": 499,
+      "irst": 500,
+      "ĠV": 501,
+      "cc": 502,
+      "Ġinc": 503,
+      "Ġ;": 504,
+      "Ġcomp": 505,
+      "ru": 506,
+      "ions": 507,
+      "Ġtheir": 508,
+      "Ġbut": 509,
+      "ide": 510,
+      "ure": 511,
+      "so": 512,
+      "Ġcont": 513,
+      "Ġint": 514,
+      "fter": 515,
+      "ical": 516,
+      "ial": 517,
+      "Ġar": 518,
+      "Ġfirst": 519,
+      "ould": 520,
+      "Ġits": 521,
+      "hed": 522,
+      "ĠâĢĵ": 523,
+      "Ġwhe": 524,
+      "wo": 525,
+      "out": 526,
+      "ub": 527,
+      "Ġ20": 528,
+      "ff": 529,
+      "Ġ:": 530,
+      "ue": 531,
+      "Ġher": 532,
+      "own": 533,
+      "ok": 534,
+      "Ġalso": 535,
+      "Ġcl": 536,
+      "per": 537,
+      "ign": 538,
+      "ater": 539,
+      "ran": 540,
+      "orm": 541,
+      "ie": 542,
+      "ome": 543,
+      "ork": 544,
+      "ass": 545,
+      "ire": 546,
+      "end": 547,
+      "Ġres": 548,
+      "Ġab": 549,
+      "Ġad": 550,
+      "Ġus": 551,
+      "ry": 552,
+      "Ġrec": 553,
+      "Ġhave": 554,
+      "age": 555,
+      "ĠHe": 556,
+      "Ġ4": 557,
+      "Ġro": 558,
+      "mer": 559,
+      "Ġone": 560,
+      "ond": 561,
+      "low": 562,
+      "Ġhas": 563,
+      "ĠTh": 564,
+      "du": 565,
+      "Ġ5": 566,
+      "Ġper": 567,
+      "Ġbeen": 568,
+      "ime": 569,
+      "Ġtwo": 570,
+      "ence": 571,
+      "land": 572,
+      "Ġ18": 573,
+      ".@": 574,
+      "Ġ@.@": 575,
+      "ult": 576,
+      "ree": 577,
+      "ough": 578,
+      "ile": 579,
+      "Ġwho": 580,
+      "ĠAl": 581,
+      "Ġsc": 582,
+      "uring": 583,
+      "pl": 584,
+      "ory": 585,
+      "ition": 586,
+      "ric": 587,
+      "ations": 588,
+      "Ġdis": 589,
+      "Ġthis": 590,
+      "Ġbec": 591,
+      "Ġapp": 592,
+      "iz": 593,
+      "ĠIt": 594,
+      "are": 595,
+      "ach": 596,
+      "lud": 597,
+      "ade": 598,
+      "Ġplay": 599,
+      "Ġj": 600,
+      "Ġman": 601,
+      "act": 602,
+      "ely": 603,
+      "Ġpart": 604,
+      "Ġdes": 605,
+      "Ġag": 606,
+      "Ġthey": 607,
+      "Ġyear": 608,
+      "ount": 609,
+      "Ġ201": 610,
+      "Ġover": 611,
+      "Ġother": 612,
+      "ound": 613,
+      "Ġafter": 614,
+      "ib": 615,
+      "over": 616,
+      "Ġser": 617,
+      "Ġen": 618,
+      "Ġoff": 619,
+      "Ġim": 620,
+      "ction": 621,
+      "ĠY": 622,
+      "ke": 623,
+      "ite": 624,
+      ",@": 625,
+      "Ġ@,@": 626,
+      "te": 627,
+      "urn": 628,
+      "Ġinclud": 629,
+      "ress": 630,
+      "ance": 631,
+      "ang": 632,
+      "Ġatt": 633,
+      "ice": 634,
+      "ace": 635,
+      "ark": 636,
+      "Ġout": 637,
+      "wn": 638,
+      "ph": 639,
+      "ember": 640,
+      "Ġpre": 641,
+      "Ġup": 642,
+      "ens": 643,
+      "man": 644,
+      "Ġev": 645,
+      "Ġtime": 646,
+      "nder": 647,
+      "rough": 648,
+      "ced": 649,
+      "Ġfin": 650,
+      "Ġinto": 651,
+      "one": 652,
+      "port": 653,
+      "round": 654,
+      "we": 655,
+      "ren": 656,
+      "les": 657,
+      "int": 658,
+      "ĠOn": 659,
+      "vel": 660,
+      "Ġcomm": 661,
+      "Ġshe": 662,
+      "ason": 663,
+      "amp": 664,
+      "Ġte": 665,
+      "Ġwould": 666,
+      "ward": 667,
+      "Ġmore": 668,
+      "Ġ6": 669,
+      "ied": 670,
+      "ose": 671,
+      "rib": 672,
+      "ĠUn": 673,
+      "Ġall": 674,
+      "ings": 675,
+      "tern": 676,
+      "ces": 677,
+      "able": 678,
+      "Ġwe": 679,
+      "ited": 680,
+      "ever": 681,
+      "ents": 682,
+      "Ġhim": 683,
+      "ased": 684,
+      "ors": 685,
+      "oy": 686,
+      "ood": 687,
+      "Ġcent": 688,
+      "ix": 689,
+      "ase": 690,
+      "ild": 691,
+      "ĠAn": 692,
+      "Ġ7": 693,
+      "Ġwork": 694,
+      "ates": 695,
+      "ious": 696,
+      "ath": 697,
+      "Ġpo": 698,
+      "rop": 699,
+      "old": 700,
+      "als": 701,
+      "iss": 702,
+      "ey": 703,
+      "ict": 704,
+      "Ġfe": 705,
+      "Ġthem": 706,
+      "gan": 707,
+      "Ġsec": 708,
+      "Ġbet": 709,
+      "Ġwhen": 710,
+      "Ġsong": 711,
+      "Ġrem": 712,
+      "ep": 713,
+      "form": 714,
+      "ail": 715,
+      "fer": 716,
+      "Ġear": 717,
+      "ubl": 718,
+      "aw": 719,
+      "Ġkn": 720,
+      "ake": 721,
+      "aus": 722,
+      "Ġmost": 723,
+      "Ġcons": 724,
+      "Ġduring": 725,
+      "ĠAs": 726,
+      "orth": 727,
+      "Ġnew": 728,
+      "ered": 729,
+      "ilm": 730,
+      "ved": 731,
+      "att": 732,
+      "Ġonly": 733,
+      "Ġ9": 734,
+      "Ġdec": 735,
+      "Ġ8": 736,
+      "ick": 737,
+      "Ġgame": 738,
+      "ons": 739,
+      "ug": 740,
+      "Ġtr": 741,
+      "ft": 742,
+      "oth": 743,
+      "ook": 744,
+      "ĠMar": 745,
+      "reat": 746,
+      "way": 747,
+      "Ġcan": 748,
+      "ollow": 749,
+      "outh": 750,
+      "ween": 751,
+      "ĠEn": 752,
+      "Ġ199": 753,
+      "ters": 754,
+      "Ġrel": 755,
+      "ind": 756,
+      "Ġabout": 757,
+      "Ġseason": 758,
+      "Ġagain": 759,
+      "ral": 760,
+      "Ġthree": 761,
+      "ational": 762,
+      "Ġunder": 763,
+      "ular": 764,
+      "Ġme": 765,
+      "Ġthan": 766,
+      "ĠCom": 767,
+      "ĠAr": 768,
+      "hip": 769,
+      "ob": 770,
+      "Ġne": 771,
+      "Ġbetween": 772,
+      "Ġfl": 773,
+      "hn": 774,
+      "ve": 775,
+      "Ġchar": 776,
+      "Ġcol": 777,
+      "Ġrecord": 778,
+      "iew": 779,
+      "ron": 780,
+      "fore": 781,
+      "Ġthrough": 782,
+      "ision": 783,
+      "orn": 784,
+      "Ġ00": 785,
+      "ock": 786,
+      "Ġver": 787,
+      "Ġlater": 788,
+      "Ġnum": 789,
+      "Ġend": 790,
+      "olog": 791,
+      "ames": 792,
+      "Ġpos": 793,
+      "Ġwrit": 794,
+      "Ġprodu": 795,
+      "Ġwhile": 796,
+      "Ġact": 797,
+      "Ġrele": 798,
+      "Ġfilm": 799,
+      "ished": 800,
+      "Ġpr": 801,
+      "ans": 802,
+      "Ġreg": 803,
+      "Ġform": 804,
+      "Ġass": 805,
+      "ĠSe": 806,
+      "ury": 807,
+      "ted": 808,
+      "ts": 809,
+      "Ġmade": 810,
+      "Ġsub": 811,
+      "Ġpe": 812,
+      "Ġso": 813,
+      "orld": 814,
+      "Ġret": 815,
+      "ĠNew": 816,
+      "Ġspec": 817,
+      "Ġacc": 818,
+      "Ġqu": 819,
+      "Ġwhere": 820,
+      "ener": 821,
+      "Ġmov": 822,
+      "hes": 823,
+      "meric": 824,
+      "ating": 825,
+      "Ġinter": 826,
+      "ĠLe": 827,
+      "ĠAmeric": 828,
+      "Ġra": 829,
+      "Ġsome": 830,
+      "Ġco": 831,
+      "Ġlar": 832,
+      "Ġbu": 833,
+      "Ġdef": 834,
+      "bum": 835,
+      "Ġac": 836,
+      "Ġmus": 837,
+      "Ġfollow": 838,
+      "ĠAt": 839,
+      "ins": 840,
+      "ived": 841,
+      "ific": 842,
+      "ual": 843,
+      "Ġam": 844,
+      "Ġsuch": 845,
+      "Ġsecond": 846,
+      "ike": 847,
+      "Ġfour": 848,
+      "Ġind": 849,
+      "ann": 850,
+      "hen": 851,
+      "Ġused": 852,
+      "ĠRe": 853,
+      "ics": 854,
+      "lect": 855,
+      "Ġday": 856,
+      "iel": 857,
+      "ily": 858,
+      "ĠThis": 859,
+      "Ġ0": 860,
+      "Ġpubl": 861,
+      "Ġcall": 862,
+      "ĠJo": 863,
+      "ll": 864,
+      "Ġalbum": 865,
+      "Ġ000": 866,
+      "rans": 867,
+      "Ġdo": 868,
+      "any": 869,
+      "Ġbefore": 870,
+      "ros": 871,
+      "ĠSh": 872,
+      "Ġsy": 873,
+      "aid": 874,
+      "ĠEng": 875,
+      "Ġbeing": 876,
+      "Ġ10": 877,
+      "uc": 878,
+      "Ġep": 879,
+      "Ġsupp": 880,
+      "Ġthere": 881,
+      "Ġyears": 882,
+      "ars": 883,
+      "owever": 884,
+      "Ġent": 885,
+      "ife": 886,
+      "Ġhigh": 887,
+      "Ġfound": 888,
+      "ird": 889,
+      "Ġno": 890,
+      "Ġset": 891,
+      "ines": 892,
+      "iver": 893,
+      "io": 894,
+      "other": 895,
+      "ject": 896,
+      "Ġsur": 897,
+      "aj": 898,
+      "ten": 899,
+      "Ġtra": 900,
+      "Ġ12": 901,
+      "ised": 902,
+      "ities": 903,
+      "velop": 904,
+      "Ġbl": 905,
+      "ale": 906,
+      "Ġseries": 907,
+      "Ġloc": 908,
+      "Ġnumber": 909,
+      "Ġpres": 910,
+      "ane": 911,
+      "ause": 912,
+      "ode": 913,
+      "ek": 914,
+      "ton": 915,
+      "ĠSc": 916,
+      "ier": 917,
+      "ise": 918,
+      "Ġsever": 919,
+      "ince": 920,
+      "Ġboth": 921,
+      "ank": 922,
+      "row": 923,
+      "irect": 924,
+      "son": 925,
+      "Ġthen": 926,
+      "ĠBrit": 927,
+      "iet": 928,
+      "Ġ16": 929,
+      "Ġepis": 930,
+      "Ġincluding": 931,
+      "its": 932,
+      "igin": 933,
+      "pr": 934,
+      "Ġ/": 935,
+      "Ġagainst": 936,
+      "Ġwell": 937,
+      "Ġbecame": 938,
+      "Ġexp": 939,
+      "Ġknown": 940,
+      "Ġtrans": 941,
+      "Ġcharac": 942,
+      "ĠâĢĶ": 943,
+      "ram": 944,
+      "Ġback": 945,
+      "Ġadd": 946,
+      "Ġpop": 947,
+      "Ġgo": 948,
+      "urch": 949,
+      "Ġdesc": 950,
+      "Ġsing": 951,
+      "ield": 952,
+      "Ġperform": 953,
+      "ained": 954,
+      "Ġrece": 955,
+      "ident": 956,
+      "Ġem": 957,
+      "ert": 958,
+      "ures": 959,
+      "Ġinv": 960,
+      "Ġdep": 961,
+      "Ġ198": 962,
+      "air": 963,
+      "ern": 964,
+      "ather": 965,
+      "ful": 966,
+      "ĠZ": 967,
+      "Ġmon": 968,
+      "Ġmany": 969,
+      "Ġmain": 970,
+      "Ġstud": 971,
+      "Ġlong": 972,
+      "inn": 973,
+      "though": 974,
+      "up": 975,
+      "ool": 976,
+      "ĠUnited": 977,
+      "led": 978,
+      "ement": 979,
+      "Ġ15": 980,
+      "ower": 981,
+      "ĠJohn": 982,
+      "Ġop": 983,
+      "Ġ11": 984,
+      "ined": 985,
+      "Ġmet": 986,
+      "ober": 987,
+      "ley": 988,
+      "Ġ17": 989,
+      "Ġcentury": 990,
+      "Ġteam": 991,
+      "Ġest": 992,
+      "ĠAfter": 993,
+      "yl": 994,
+      "Ġmin": 995,
+      "uch": 996,
+      "ute": 997,
+      "Ġdevelop": 998,
+      "ĠShe": 999,
+      "iam": 1000,
+      "Ġshow": 1001,
+      "elf": 1002,
+      "Ġrep": 1003,
+      "Ġconc": 1004,
+      "ative": 1005,
+      "Ġcre": 1006,
+      "overn": 1007,
+      "ared": 1008,
+      "Ġ194": 1009,
+      "Ġorigin": 1010,
+      "Ġsm": 1011,
+      "ivers": 1012,
+      "az": 1013,
+      "Ġlead": 1014,
+      "Ġseveral": 1015,
+      "ah": 1016,
+      "Ġob": 1017,
+      "Ġrev": 1018,
+      "Ġmill": 1019,
+      "erm": 1020,
+      "ually": 1021,
+      "oot": 1022,
+      "Ġbegan": 1023
+    },
+    "merges": [
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        ","
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "Ġ",
+        "."
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "a",
+        "s"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "i",
+        "on"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "="
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "Ġ",
+        "1"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "Ġ",
+        "C"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "Ġ",
+        "Ċ"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "Ġ",
+        "@"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "a",
+        "c"
+      ],
+      [
+        "Ġw",
+        "as"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "ĠT",
+        "he"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "Ġ",
+        "on"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "Ġ",
+        "'"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "o",
+        "l"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "i",
+        "v"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "-",
+        "@"
+      ],
+      [
+        "Ġ@",
+        "-@"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "u",
+        "s"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "Ġ",
+        "2"
+      ],
+      [
+        "Ġ",
+        "P"
+      ],
+      [
+        "at",
+        "ion"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġ",
+        "R"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "t",
+        "h"
+      ],
+      [
+        "Ġ",
+        "D"
+      ],
+      [
+        "Ġ",
+        "is"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "e",
+        "m"
+      ],
+      [
+        "Ġ",
+        "F"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "u",
+        "l"
+      ],
+      [
+        "Ġb",
+        "y"
+      ],
+      [
+        "Ġa",
+        "l"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "Ġ",
+        ")"
+      ],
+      [
+        "Ġ",
+        "("
+      ],
+      [
+        "Ġ",
+        "W"
+      ],
+      [
+        "Ġc",
+        "on"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "Ġ",
+        "G"
+      ],
+      [
+        "o",
+        "s"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġ",
+        "N"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "er",
+        "s"
+      ],
+      [
+        "c",
+        "t"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "Ġ1",
+        "9"
+      ],
+      [
+        "ro",
+        "m"
+      ],
+      [
+        "a",
+        "nd"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "es",
+        "t"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "a",
+        "g"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "0",
+        "0"
+      ],
+      [
+        "is",
+        "t"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "o",
+        "d"
+      ],
+      [
+        "a",
+        "v"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "Ġ",
+        "O"
+      ],
+      [
+        "Ġf",
+        "rom"
+      ],
+      [
+        "Ġc",
+        "om"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "o",
+        "p"
+      ],
+      [
+        "Ġp",
+        "ro"
+      ],
+      [
+        "re",
+        "s"
+      ],
+      [
+        "i",
+        "es"
+      ],
+      [
+        "i",
+        "f"
+      ],
+      [
+        "Ġ",
+        "v"
+      ],
+      [
+        "or",
+        "t"
+      ],
+      [
+        "er",
+        "e"
+      ],
+      [
+        "il",
+        "l"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "Ġd",
+        "e"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "Ġs",
+        "u"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "ĠI",
+        "n"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġw",
+        "ere"
+      ],
+      [
+        "e",
+        "w"
+      ],
+      [
+        "on",
+        "g"
+      ],
+      [
+        "ig",
+        "h"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "at",
+        "e"
+      ],
+      [
+        "al",
+        "l"
+      ],
+      [
+        "ar",
+        "t"
+      ],
+      [
+        "a",
+        "k"
+      ],
+      [
+        "ic",
+        "h"
+      ],
+      [
+        "Ġc",
+        "h"
+      ],
+      [
+        "Ġo",
+        "r"
+      ],
+      [
+        "a",
+        "b"
+      ],
+      [
+        "an",
+        "t"
+      ],
+      [
+        "u",
+        "d"
+      ],
+      [
+        "o",
+        "c"
+      ],
+      [
+        "b",
+        "er"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "g",
+        "h"
+      ],
+      [
+        "it",
+        "y"
+      ],
+      [
+        "at",
+        "ed"
+      ],
+      [
+        "p",
+        "t"
+      ],
+      [
+        "es",
+        "s"
+      ],
+      [
+        "e",
+        "ar"
+      ],
+      [
+        "Ġ",
+        "K"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "q",
+        "u"
+      ],
+      [
+        "iv",
+        "e"
+      ],
+      [
+        "ro",
+        "u"
+      ],
+      [
+        "Ġa",
+        "re"
+      ],
+      [
+        "Ġ",
+        "â"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "ac",
+        "k"
+      ],
+      [
+        "ec",
+        "t"
+      ],
+      [
+        "Ġâ",
+        "Ģ"
+      ],
+      [
+        "Ġ",
+        "U"
+      ],
+      [
+        "Ġh",
+        "ad"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġwh",
+        "ich"
+      ],
+      [
+        "re",
+        "d"
+      ],
+      [
+        "o",
+        "v"
+      ],
+      [
+        "ĠS",
+        "t"
+      ],
+      [
+        "as",
+        "t"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "i",
+        "an"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "m",
+        "ent"
+      ],
+      [
+        "Ġ",
+        "le"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "g",
+        "e"
+      ],
+      [
+        "or",
+        "d"
+      ],
+      [
+        "r",
+        "it"
+      ],
+      [
+        "i",
+        "p"
+      ],
+      [
+        "in",
+        "e"
+      ],
+      [
+        "el",
+        "l"
+      ],
+      [
+        "al",
+        "ly"
+      ],
+      [
+        "ou",
+        "r"
+      ],
+      [
+        "o",
+        "st"
+      ],
+      [
+        "igh",
+        "t"
+      ],
+      [
+        "t",
+        "her"
+      ],
+      [
+        "a",
+        "p"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "is",
+        "h"
+      ],
+      [
+        "ĠC",
+        "h"
+      ],
+      [
+        "ou",
+        "n"
+      ],
+      [
+        "i",
+        "a"
+      ],
+      [
+        "Ġ",
+        "3"
+      ],
+      [
+        "av",
+        "e"
+      ],
+      [
+        "ar",
+        "y"
+      ],
+      [
+        "u",
+        "st"
+      ],
+      [
+        "o",
+        "g"
+      ],
+      [
+        "Ġ2",
+        "00"
+      ],
+      [
+        "Ġ",
+        "un"
+      ],
+      [
+        "ou",
+        "s"
+      ],
+      [
+        "ir",
+        "st"
+      ],
+      [
+        "Ġ",
+        "V"
+      ],
+      [
+        "c",
+        "c"
+      ],
+      [
+        "Ġin",
+        "c"
+      ],
+      [
+        "Ġ",
+        ";"
+      ],
+      [
+        "Ġcom",
+        "p"
+      ],
+      [
+        "r",
+        "u"
+      ],
+      [
+        "ion",
+        "s"
+      ],
+      [
+        "Ġthe",
+        "ir"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "u",
+        "re"
+      ],
+      [
+        "s",
+        "o"
+      ],
+      [
+        "Ġcon",
+        "t"
+      ],
+      [
+        "Ġin",
+        "t"
+      ],
+      [
+        "f",
+        "ter"
+      ],
+      [
+        "ic",
+        "al"
+      ],
+      [
+        "i",
+        "al"
+      ],
+      [
+        "Ġa",
+        "r"
+      ],
+      [
+        "Ġf",
+        "irst"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "Ġit",
+        "s"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "ĠâĢ",
+        "ĵ"
+      ],
+      [
+        "Ġw",
+        "he"
+      ],
+      [
+        "w",
+        "o"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "u",
+        "b"
+      ],
+      [
+        "Ġ2",
+        "0"
+      ],
+      [
+        "f",
+        "f"
+      ],
+      [
+        "Ġ",
+        ":"
+      ],
+      [
+        "u",
+        "e"
+      ],
+      [
+        "Ġ",
+        "her"
+      ],
+      [
+        "ow",
+        "n"
+      ],
+      [
+        "o",
+        "k"
+      ],
+      [
+        "Ġal",
+        "so"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "p",
+        "er"
+      ],
+      [
+        "ig",
+        "n"
+      ],
+      [
+        "at",
+        "er"
+      ],
+      [
+        "r",
+        "an"
+      ],
+      [
+        "or",
+        "m"
+      ],
+      [
+        "i",
+        "e"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "or",
+        "k"
+      ],
+      [
+        "as",
+        "s"
+      ],
+      [
+        "i",
+        "re"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "Ġre",
+        "s"
+      ],
+      [
+        "Ġa",
+        "b"
+      ],
+      [
+        "Ġa",
+        "d"
+      ],
+      [
+        "Ġ",
+        "us"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        "Ġre",
+        "c"
+      ],
+      [
+        "Ġh",
+        "ave"
+      ],
+      [
+        "ag",
+        "e"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "Ġ",
+        "4"
+      ],
+      [
+        "Ġ",
+        "ro"
+      ],
+      [
+        "m",
+        "er"
+      ],
+      [
+        "Ġon",
+        "e"
+      ],
+      [
+        "on",
+        "d"
+      ],
+      [
+        "l",
+        "ow"
+      ],
+      [
+        "Ġh",
+        "as"
+      ],
+      [
+        "ĠT",
+        "h"
+      ],
+      [
+        "d",
+        "u"
+      ],
+      [
+        "Ġ",
+        "5"
+      ],
+      [
+        "Ġp",
+        "er"
+      ],
+      [
+        "Ġbe",
+        "en"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "Ġt",
+        "wo"
+      ],
+      [
+        "en",
+        "ce"
+      ],
+      [
+        "l",
+        "and"
+      ],
+      [
+        "Ġ1",
+        "8"
+      ],
+      [
+        ".",
+        "@"
+      ],
+      [
+        "Ġ@",
+        ".@"
+      ],
+      [
+        "ul",
+        "t"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "ou",
+        "gh"
+      ],
+      [
+        "i",
+        "le"
+      ],
+      [
+        "Ġwh",
+        "o"
+      ],
+      [
+        "ĠA",
+        "l"
+      ],
+      [
+        "Ġs",
+        "c"
+      ],
+      [
+        "ur",
+        "ing"
+      ],
+      [
+        "p",
+        "l"
+      ],
+      [
+        "or",
+        "y"
+      ],
+      [
+        "it",
+        "ion"
+      ],
+      [
+        "r",
+        "ic"
+      ],
+      [
+        "ation",
+        "s"
+      ],
+      [
+        "Ġd",
+        "is"
+      ],
+      [
+        "Ġth",
+        "is"
+      ],
+      [
+        "Ġb",
+        "ec"
+      ],
+      [
+        "Ġa",
+        "pp"
+      ],
+      [
+        "i",
+        "z"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "ac",
+        "h"
+      ],
+      [
+        "l",
+        "ud"
+      ],
+      [
+        "ad",
+        "e"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "Ġm",
+        "an"
+      ],
+      [
+        "ac",
+        "t"
+      ],
+      [
+        "el",
+        "y"
+      ],
+      [
+        "Ġp",
+        "art"
+      ],
+      [
+        "Ġd",
+        "es"
+      ],
+      [
+        "Ġa",
+        "g"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "Ġy",
+        "ear"
+      ],
+      [
+        "oun",
+        "t"
+      ],
+      [
+        "Ġ20",
+        "1"
+      ],
+      [
+        "Ġo",
+        "ver"
+      ],
+      [
+        "Ġo",
+        "ther"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "Ġa",
+        "fter"
+      ],
+      [
+        "i",
+        "b"
+      ],
+      [
+        "o",
+        "ver"
+      ],
+      [
+        "Ġs",
+        "er"
+      ],
+      [
+        "Ġ",
+        "en"
+      ],
+      [
+        "Ġof",
+        "f"
+      ],
+      [
+        "Ġ",
+        "im"
+      ],
+      [
+        "ct",
+        "ion"
+      ],
+      [
+        "Ġ",
+        "Y"
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "it",
+        "e"
+      ],
+      [
+        ",",
+        "@"
+      ],
+      [
+        "Ġ@",
+        ",@"
+      ],
+      [
+        "t",
+        "e"
+      ],
+      [
+        "ur",
+        "n"
+      ],
+      [
+        "Ġinc",
+        "lud"
+      ],
+      [
+        "res",
+        "s"
+      ],
+      [
+        "an",
+        "ce"
+      ],
+      [
+        "an",
+        "g"
+      ],
+      [
+        "Ġat",
+        "t"
+      ],
+      [
+        "ic",
+        "e"
+      ],
+      [
+        "ac",
+        "e"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "Ġo",
+        "ut"
+      ],
+      [
+        "w",
+        "n"
+      ],
+      [
+        "p",
+        "h"
+      ],
+      [
+        "em",
+        "ber"
+      ],
+      [
+        "Ġp",
+        "re"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "en",
+        "s"
+      ],
+      [
+        "m",
+        "an"
+      ],
+      [
+        "Ġe",
+        "v"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "nd",
+        "er"
+      ],
+      [
+        "rou",
+        "gh"
+      ],
+      [
+        "c",
+        "ed"
+      ],
+      [
+        "Ġf",
+        "in"
+      ],
+      [
+        "Ġint",
+        "o"
+      ],
+      [
+        "on",
+        "e"
+      ],
+      [
+        "p",
+        "ort"
+      ],
+      [
+        "rou",
+        "nd"
+      ],
+      [
+        "w",
+        "e"
+      ],
+      [
+        "re",
+        "n"
+      ],
+      [
+        "l",
+        "es"
+      ],
+      [
+        "in",
+        "t"
+      ],
+      [
+        "ĠO",
+        "n"
+      ],
+      [
+        "v",
+        "el"
+      ],
+      [
+        "Ġcom",
+        "m"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "as",
+        "on"
+      ],
+      [
+        "am",
+        "p"
+      ],
+      [
+        "Ġt",
+        "e"
+      ],
+      [
+        "Ġw",
+        "ould"
+      ],
+      [
+        "w",
+        "ard"
+      ],
+      [
+        "Ġm",
+        "ore"
+      ],
+      [
+        "Ġ",
+        "6"
+      ],
+      [
+        "i",
+        "ed"
+      ],
+      [
+        "os",
+        "e"
+      ],
+      [
+        "ri",
+        "b"
+      ],
+      [
+        "ĠU",
+        "n"
+      ],
+      [
+        "Ġal",
+        "l"
+      ],
+      [
+        "ing",
+        "s"
+      ],
+      [
+        "ter",
+        "n"
+      ],
+      [
+        "c",
+        "es"
+      ],
+      [
+        "ab",
+        "le"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "it",
+        "ed"
+      ],
+      [
+        "e",
+        "ver"
+      ],
+      [
+        "ent",
+        "s"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "as",
+        "ed"
+      ],
+      [
+        "or",
+        "s"
+      ],
+      [
+        "o",
+        "y"
+      ],
+      [
+        "o",
+        "od"
+      ],
+      [
+        "Ġc",
+        "ent"
+      ],
+      [
+        "i",
+        "x"
+      ],
+      [
+        "as",
+        "e"
+      ],
+      [
+        "il",
+        "d"
+      ],
+      [
+        "ĠA",
+        "n"
+      ],
+      [
+        "Ġ",
+        "7"
+      ],
+      [
+        "Ġw",
+        "ork"
+      ],
+      [
+        "at",
+        "es"
+      ],
+      [
+        "i",
+        "ous"
+      ],
+      [
+        "at",
+        "h"
+      ],
+      [
+        "Ġp",
+        "o"
+      ],
+      [
+        "ro",
+        "p"
+      ],
+      [
+        "ol",
+        "d"
+      ],
+      [
+        "al",
+        "s"
+      ],
+      [
+        "is",
+        "s"
+      ],
+      [
+        "e",
+        "y"
+      ],
+      [
+        "ic",
+        "t"
+      ],
+      [
+        "Ġf",
+        "e"
+      ],
+      [
+        "Ġthe",
+        "m"
+      ],
+      [
+        "g",
+        "an"
+      ],
+      [
+        "Ġs",
+        "ec"
+      ],
+      [
+        "Ġb",
+        "et"
+      ],
+      [
+        "Ġwhe",
+        "n"
+      ],
+      [
+        "Ġs",
+        "ong"
+      ],
+      [
+        "Ġre",
+        "m"
+      ],
+      [
+        "e",
+        "p"
+      ],
+      [
+        "f",
+        "orm"
+      ],
+      [
+        "a",
+        "il"
+      ],
+      [
+        "f",
+        "er"
+      ],
+      [
+        "Ġe",
+        "ar"
+      ],
+      [
+        "ub",
+        "l"
+      ],
+      [
+        "a",
+        "w"
+      ],
+      [
+        "Ġk",
+        "n"
+      ],
+      [
+        "ak",
+        "e"
+      ],
+      [
+        "a",
+        "us"
+      ],
+      [
+        "Ġm",
+        "ost"
+      ],
+      [
+        "Ġcon",
+        "s"
+      ],
+      [
+        "Ġd",
+        "uring"
+      ],
+      [
+        "ĠA",
+        "s"
+      ],
+      [
+        "or",
+        "th"
+      ],
+      [
+        "Ġn",
+        "ew"
+      ],
+      [
+        "er",
+        "ed"
+      ],
+      [
+        "il",
+        "m"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        "at",
+        "t"
+      ],
+      [
+        "Ġon",
+        "ly"
+      ],
+      [
+        "Ġ",
+        "9"
+      ],
+      [
+        "Ġd",
+        "ec"
+      ],
+      [
+        "Ġ",
+        "8"
+      ],
+      [
+        "ic",
+        "k"
+      ],
+      [
+        "Ġg",
+        "ame"
+      ],
+      [
+        "on",
+        "s"
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "Ġt",
+        "r"
+      ],
+      [
+        "f",
+        "t"
+      ],
+      [
+        "ot",
+        "h"
+      ],
+      [
+        "o",
+        "ok"
+      ],
+      [
+        "ĠM",
+        "ar"
+      ],
+      [
+        "re",
+        "at"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "ol",
+        "low"
+      ],
+      [
+        "ou",
+        "th"
+      ],
+      [
+        "we",
+        "en"
+      ],
+      [
+        "ĠE",
+        "n"
+      ],
+      [
+        "Ġ19",
+        "9"
+      ],
+      [
+        "ter",
+        "s"
+      ],
+      [
+        "Ġre",
+        "l"
+      ],
+      [
+        "in",
+        "d"
+      ],
+      [
+        "Ġab",
+        "out"
+      ],
+      [
+        "Ġse",
+        "ason"
+      ],
+      [
+        "Ġag",
+        "ain"
+      ],
+      [
+        "r",
+        "al"
+      ],
+      [
+        "Ġth",
+        "ree"
+      ],
+      [
+        "ation",
+        "al"
+      ],
+      [
+        "Ġu",
+        "nder"
+      ],
+      [
+        "ul",
+        "ar"
+      ],
+      [
+        "Ġm",
+        "e"
+      ],
+      [
+        "Ġth",
+        "an"
+      ],
+      [
+        "ĠC",
+        "om"
+      ],
+      [
+        "ĠA",
+        "r"
+      ],
+      [
+        "h",
+        "ip"
+      ],
+      [
+        "o",
+        "b"
+      ],
+      [
+        "Ġn",
+        "e"
+      ],
+      [
+        "Ġbet",
+        "ween"
+      ],
+      [
+        "Ġf",
+        "l"
+      ],
+      [
+        "h",
+        "n"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġch",
+        "ar"
+      ],
+      [
+        "Ġc",
+        "ol"
+      ],
+      [
+        "Ġrec",
+        "ord"
+      ],
+      [
+        "i",
+        "ew"
+      ],
+      [
+        "r",
+        "on"
+      ],
+      [
+        "f",
+        "ore"
+      ],
+      [
+        "Ġth",
+        "rough"
+      ],
+      [
+        "is",
+        "ion"
+      ],
+      [
+        "or",
+        "n"
+      ],
+      [
+        "Ġ",
+        "00"
+      ],
+      [
+        "oc",
+        "k"
+      ],
+      [
+        "Ġ",
+        "ver"
+      ],
+      [
+        "Ġl",
+        "ater"
+      ],
+      [
+        "Ġn",
+        "um"
+      ],
+      [
+        "Ġe",
+        "nd"
+      ],
+      [
+        "ol",
+        "og"
+      ],
+      [
+        "am",
+        "es"
+      ],
+      [
+        "Ġp",
+        "os"
+      ],
+      [
+        "Ġw",
+        "rit"
+      ],
+      [
+        "Ġpro",
+        "du"
+      ],
+      [
+        "Ġwh",
+        "ile"
+      ],
+      [
+        "Ġa",
+        "ct"
+      ],
+      [
+        "Ġre",
+        "le"
+      ],
+      [
+        "Ġf",
+        "ilm"
+      ],
+      [
+        "is",
+        "hed"
+      ],
+      [
+        "Ġp",
+        "r"
+      ],
+      [
+        "an",
+        "s"
+      ],
+      [
+        "Ġre",
+        "g"
+      ],
+      [
+        "Ġfor",
+        "m"
+      ],
+      [
+        "Ġas",
+        "s"
+      ],
+      [
+        "ĠS",
+        "e"
+      ],
+      [
+        "ur",
+        "y"
+      ],
+      [
+        "t",
+        "ed"
+      ],
+      [
+        "t",
+        "s"
+      ],
+      [
+        "Ġm",
+        "ade"
+      ],
+      [
+        "Ġsu",
+        "b"
+      ],
+      [
+        "Ġp",
+        "e"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "or",
+        "ld"
+      ],
+      [
+        "Ġre",
+        "t"
+      ],
+      [
+        "ĠN",
+        "ew"
+      ],
+      [
+        "Ġsp",
+        "ec"
+      ],
+      [
+        "Ġa",
+        "cc"
+      ],
+      [
+        "Ġ",
+        "qu"
+      ],
+      [
+        "Ġwhe",
+        "re"
+      ],
+      [
+        "en",
+        "er"
+      ],
+      [
+        "Ġm",
+        "ov"
+      ],
+      [
+        "he",
+        "s"
+      ],
+      [
+        "mer",
+        "ic"
+      ],
+      [
+        "at",
+        "ing"
+      ],
+      [
+        "Ġin",
+        "ter"
+      ],
+      [
+        "ĠL",
+        "e"
+      ],
+      [
+        "ĠA",
+        "meric"
+      ],
+      [
+        "Ġ",
+        "ra"
+      ],
+      [
+        "Ġs",
+        "ome"
+      ],
+      [
+        "Ġc",
+        "o"
+      ],
+      [
+        "Ġl",
+        "ar"
+      ],
+      [
+        "Ġb",
+        "u"
+      ],
+      [
+        "Ġde",
+        "f"
+      ],
+      [
+        "b",
+        "um"
+      ],
+      [
+        "Ġa",
+        "c"
+      ],
+      [
+        "Ġm",
+        "us"
+      ],
+      [
+        "Ġf",
+        "ollow"
+      ],
+      [
+        "ĠA",
+        "t"
+      ],
+      [
+        "in",
+        "s"
+      ],
+      [
+        "iv",
+        "ed"
+      ],
+      [
+        "if",
+        "ic"
+      ],
+      [
+        "u",
+        "al"
+      ],
+      [
+        "Ġa",
+        "m"
+      ],
+      [
+        "Ġsu",
+        "ch"
+      ],
+      [
+        "Ġsec",
+        "ond"
+      ],
+      [
+        "i",
+        "ke"
+      ],
+      [
+        "Ġf",
+        "our"
+      ],
+      [
+        "Ġin",
+        "d"
+      ],
+      [
+        "an",
+        "n"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "Ġus",
+        "ed"
+      ],
+      [
+        "ĠR",
+        "e"
+      ],
+      [
+        "ic",
+        "s"
+      ],
+      [
+        "le",
+        "ct"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "i",
+        "el"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "ĠTh",
+        "is"
+      ],
+      [
+        "Ġ",
+        "0"
+      ],
+      [
+        "Ġp",
+        "ubl"
+      ],
+      [
+        "Ġc",
+        "all"
+      ],
+      [
+        "ĠJ",
+        "o"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "Ġal",
+        "bum"
+      ],
+      [
+        "Ġ00",
+        "0"
+      ],
+      [
+        "ran",
+        "s"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "an",
+        "y"
+      ],
+      [
+        "Ġbe",
+        "fore"
+      ],
+      [
+        "ro",
+        "s"
+      ],
+      [
+        "ĠS",
+        "h"
+      ],
+      [
+        "Ġs",
+        "y"
+      ],
+      [
+        "a",
+        "id"
+      ],
+      [
+        "ĠEn",
+        "g"
+      ],
+      [
+        "Ġbe",
+        "ing"
+      ],
+      [
+        "Ġ1",
+        "0"
+      ],
+      [
+        "u",
+        "c"
+      ],
+      [
+        "Ġe",
+        "p"
+      ],
+      [
+        "Ġsu",
+        "pp"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "Ġyear",
+        "s"
+      ],
+      [
+        "ar",
+        "s"
+      ],
+      [
+        "ow",
+        "ever"
+      ],
+      [
+        "Ġ",
+        "ent"
+      ],
+      [
+        "if",
+        "e"
+      ],
+      [
+        "Ġh",
+        "igh"
+      ],
+      [
+        "Ġf",
+        "ound"
+      ],
+      [
+        "ir",
+        "d"
+      ],
+      [
+        "Ġn",
+        "o"
+      ],
+      [
+        "Ġs",
+        "et"
+      ],
+      [
+        "in",
+        "es"
+      ],
+      [
+        "iv",
+        "er"
+      ],
+      [
+        "i",
+        "o"
+      ],
+      [
+        "ot",
+        "her"
+      ],
+      [
+        "j",
+        "ect"
+      ],
+      [
+        "Ġs",
+        "ur"
+      ],
+      [
+        "a",
+        "j"
+      ],
+      [
+        "t",
+        "en"
+      ],
+      [
+        "Ġt",
+        "ra"
+      ],
+      [
+        "Ġ1",
+        "2"
+      ],
+      [
+        "is",
+        "ed"
+      ],
+      [
+        "it",
+        "ies"
+      ],
+      [
+        "vel",
+        "op"
+      ],
+      [
+        "Ġb",
+        "l"
+      ],
+      [
+        "al",
+        "e"
+      ],
+      [
+        "Ġser",
+        "ies"
+      ],
+      [
+        "Ġl",
+        "oc"
+      ],
+      [
+        "Ġnum",
+        "ber"
+      ],
+      [
+        "Ġp",
+        "res"
+      ],
+      [
+        "an",
+        "e"
+      ],
+      [
+        "aus",
+        "e"
+      ],
+      [
+        "od",
+        "e"
+      ],
+      [
+        "e",
+        "k"
+      ],
+      [
+        "t",
+        "on"
+      ],
+      [
+        "ĠS",
+        "c"
+      ],
+      [
+        "i",
+        "er"
+      ],
+      [
+        "is",
+        "e"
+      ],
+      [
+        "Ġse",
+        "ver"
+      ],
+      [
+        "in",
+        "ce"
+      ],
+      [
+        "Ġb",
+        "oth"
+      ],
+      [
+        "an",
+        "k"
+      ],
+      [
+        "ro",
+        "w"
+      ],
+      [
+        "ire",
+        "ct"
+      ],
+      [
+        "s",
+        "on"
+      ],
+      [
+        "Ġthe",
+        "n"
+      ],
+      [
+        "ĠB",
+        "rit"
+      ],
+      [
+        "i",
+        "et"
+      ],
+      [
+        "Ġ1",
+        "6"
+      ],
+      [
+        "Ġep",
+        "is"
+      ],
+      [
+        "Ġinclud",
+        "ing"
+      ],
+      [
+        "it",
+        "s"
+      ],
+      [
+        "ig",
+        "in"
+      ],
+      [
+        "p",
+        "r"
+      ],
+      [
+        "Ġ",
+        "/"
+      ],
+      [
+        "Ġagain",
+        "st"
+      ],
+      [
+        "Ġw",
+        "ell"
+      ],
+      [
+        "Ġbec",
+        "ame"
+      ],
+      [
+        "Ġex",
+        "p"
+      ],
+      [
+        "Ġkn",
+        "own"
+      ],
+      [
+        "Ġt",
+        "rans"
+      ],
+      [
+        "Ġchar",
+        "ac"
+      ],
+      [
+        "ĠâĢ",
+        "Ķ"
+      ],
+      [
+        "r",
+        "am"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "Ġad",
+        "d"
+      ],
+      [
+        "Ġp",
+        "op"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "ur",
+        "ch"
+      ],
+      [
+        "Ġdes",
+        "c"
+      ],
+      [
+        "Ġs",
+        "ing"
+      ],
+      [
+        "iel",
+        "d"
+      ],
+      [
+        "Ġper",
+        "form"
+      ],
+      [
+        "ain",
+        "ed"
+      ],
+      [
+        "Ġre",
+        "ce"
+      ],
+      [
+        "id",
+        "ent"
+      ],
+      [
+        "Ġe",
+        "m"
+      ],
+      [
+        "er",
+        "t"
+      ],
+      [
+        "u",
+        "res"
+      ],
+      [
+        "Ġin",
+        "v"
+      ],
+      [
+        "Ġde",
+        "p"
+      ],
+      [
+        "Ġ19",
+        "8"
+      ],
+      [
+        "a",
+        "ir"
+      ],
+      [
+        "er",
+        "n"
+      ],
+      [
+        "at",
+        "her"
+      ],
+      [
+        "f",
+        "ul"
+      ],
+      [
+        "Ġ",
+        "Z"
+      ],
+      [
+        "Ġm",
+        "on"
+      ],
+      [
+        "Ġman",
+        "y"
+      ],
+      [
+        "Ġm",
+        "ain"
+      ],
+      [
+        "Ġst",
+        "ud"
+      ],
+      [
+        "Ġl",
+        "ong"
+      ],
+      [
+        "in",
+        "n"
+      ],
+      [
+        "th",
+        "ough"
+      ],
+      [
+        "u",
+        "p"
+      ],
+      [
+        "o",
+        "ol"
+      ],
+      [
+        "ĠUn",
+        "ited"
+      ],
+      [
+        "l",
+        "ed"
+      ],
+      [
+        "em",
+        "ent"
+      ],
+      [
+        "Ġ1",
+        "5"
+      ],
+      [
+        "ow",
+        "er"
+      ],
+      [
+        "ĠJo",
+        "hn"
+      ],
+      [
+        "Ġo",
+        "p"
+      ],
+      [
+        "Ġ1",
+        "1"
+      ],
+      [
+        "in",
+        "ed"
+      ],
+      [
+        "Ġm",
+        "et"
+      ],
+      [
+        "o",
+        "ber"
+      ],
+      [
+        "le",
+        "y"
+      ],
+      [
+        "Ġ1",
+        "7"
+      ],
+      [
+        "Ġcent",
+        "ury"
+      ],
+      [
+        "Ġte",
+        "am"
+      ],
+      [
+        "Ġ",
+        "est"
+      ],
+      [
+        "ĠA",
+        "fter"
+      ],
+      [
+        "y",
+        "l"
+      ],
+      [
+        "Ġm",
+        "in"
+      ],
+      [
+        "u",
+        "ch"
+      ],
+      [
+        "ut",
+        "e"
+      ],
+      [
+        "Ġde",
+        "velop"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "i",
+        "am"
+      ],
+      [
+        "Ġsh",
+        "ow"
+      ],
+      [
+        "el",
+        "f"
+      ],
+      [
+        "Ġre",
+        "p"
+      ],
+      [
+        "Ġcon",
+        "c"
+      ],
+      [
+        "at",
+        "ive"
+      ],
+      [
+        "Ġc",
+        "re"
+      ],
+      [
+        "over",
+        "n"
+      ],
+      [
+        "a",
+        "red"
+      ],
+      [
+        "Ġ19",
+        "4"
+      ],
+      [
+        "Ġor",
+        "igin"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "iv",
+        "ers"
+      ],
+      [
+        "a",
+        "z"
+      ],
+      [
+        "Ġle",
+        "ad"
+      ],
+      [
+        "Ġsever",
+        "al"
+      ],
+      [
+        "a",
+        "h"
+      ],
+      [
+        "Ġo",
+        "b"
+      ],
+      [
+        "Ġre",
+        "v"
+      ],
+      [
+        "Ġm",
+        "ill"
+      ],
+      [
+        "er",
+        "m"
+      ],
+      [
+        "u",
+        "ally"
+      ],
+      [
+        "o",
+        "ot"
+      ],
+      [
+        "Ġbe",
+        "gan"
+      ]
+    ]
+  }
+}

last-checkpoint/tokenizer_config.json

@@ -0,0 +1,39 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1024": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

last-checkpoint/trainer_state.json

@@ -0,0 +1,233 @@
+{
+  "best_metric": 6.917991638183594,
+  "best_model_checkpoint": "miner_id_24/checkpoint-25",
+  "epoch": 0.011291779584462511,
+  "eval_steps": 25,
+  "global_step": 25,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00045167118337850043,
+      "grad_norm": 0.09511399269104004,
+      "learning_rate": 0.0001,
+      "loss": 6.9411,
+      "step": 1
+    },
+    {
+      "epoch": 0.00045167118337850043,
+      "eval_loss": 6.941762447357178,
+      "eval_runtime": 4.1373,
+      "eval_samples_per_second": 901.316,
+      "eval_steps_per_second": 112.876,
+      "step": 1
+    },
+    {
+      "epoch": 0.0009033423667570009,
+      "grad_norm": 0.12681150436401367,
+      "learning_rate": 0.0002,
+      "loss": 6.9449,
+      "step": 2
+    },
+    {
+      "epoch": 0.0013550135501355014,
+      "grad_norm": 0.14625313878059387,
+      "learning_rate": 0.00019978589232386035,
+      "loss": 6.9428,
+      "step": 3
+    },
+    {
+      "epoch": 0.0018066847335140017,
+      "grad_norm": 0.16340851783752441,
+      "learning_rate": 0.00019914448613738106,
+      "loss": 6.9417,
+      "step": 4
+    },
+    {
+      "epoch": 0.002258355916892502,
+      "grad_norm": 0.18675823509693146,
+      "learning_rate": 0.00019807852804032305,
+      "loss": 6.9435,
+      "step": 5
+    },
+    {
+      "epoch": 0.0027100271002710027,
+      "grad_norm": 0.16288436949253082,
+      "learning_rate": 0.00019659258262890683,
+      "loss": 6.9424,
+      "step": 6
+    },
+    {
+      "epoch": 0.0031616982836495033,
+      "grad_norm": 0.19281183183193207,
+      "learning_rate": 0.0001946930129495106,
+      "loss": 6.9353,
+      "step": 7
+    },
+    {
+      "epoch": 0.0036133694670280035,
+      "grad_norm": 0.18037211894989014,
+      "learning_rate": 0.0001923879532511287,
+      "loss": 6.9329,
+      "step": 8
+    },
+    {
+      "epoch": 0.0040650406504065045,
+      "grad_norm": 0.18146973848342896,
+      "learning_rate": 0.00018968727415326884,
+      "loss": 6.9382,
+      "step": 9
+    },
+    {
+      "epoch": 0.004516711833785004,
+      "grad_norm": 0.2021256536245346,
+      "learning_rate": 0.00018660254037844388,
+      "loss": 6.9314,
+      "step": 10
+    },
+    {
+      "epoch": 0.004968383017163505,
+      "grad_norm": 0.20851069688796997,
+      "learning_rate": 0.00018314696123025454,
+      "loss": 6.9346,
+      "step": 11
+    },
+    {
+      "epoch": 0.005420054200542005,
+      "grad_norm": 0.20332486927509308,
+      "learning_rate": 0.00017933533402912354,
+      "loss": 6.9377,
+      "step": 12
+    },
+    {
+      "epoch": 0.005871725383920506,
+      "grad_norm": 0.11624668538570404,
+      "learning_rate": 0.00017518398074789775,
+      "loss": 6.9365,
+      "step": 13
+    },
+    {
+      "epoch": 0.006323396567299007,
+      "grad_norm": 0.13349321484565735,
+      "learning_rate": 0.00017071067811865476,
+      "loss": 6.9301,
+      "step": 14
+    },
+    {
+      "epoch": 0.006775067750677507,
+      "grad_norm": 0.1457970142364502,
+      "learning_rate": 0.00016593458151000688,
+      "loss": 6.9293,
+      "step": 15
+    },
+    {
+      "epoch": 0.007226738934056007,
+      "grad_norm": 0.1603357046842575,
+      "learning_rate": 0.00016087614290087208,
+      "loss": 6.9314,
+      "step": 16
+    },
+    {
+      "epoch": 0.0076784101174345075,
+      "grad_norm": 0.16457444429397583,
+      "learning_rate": 0.00015555702330196023,
+      "loss": 6.9303,
+      "step": 17
+    },
+    {
+      "epoch": 0.008130081300813009,
+      "grad_norm": 0.20100098848342896,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 6.9251,
+      "step": 18
+    },
+    {
+      "epoch": 0.008581752484191508,
+      "grad_norm": 0.18849341571331024,
+      "learning_rate": 0.00014422886902190014,
+      "loss": 6.9223,
+      "step": 19
+    },
+    {
+      "epoch": 0.009033423667570008,
+      "grad_norm": 0.18996189534664154,
+      "learning_rate": 0.000138268343236509,
+      "loss": 6.9237,
+      "step": 20
+    },
+    {
+      "epoch": 0.009485094850948509,
+      "grad_norm": 0.1919945478439331,
+      "learning_rate": 0.00013214394653031616,
+      "loss": 6.9267,
+      "step": 21
+    },
+    {
+      "epoch": 0.00993676603432701,
+      "grad_norm": 0.2134779840707779,
+      "learning_rate": 0.00012588190451025207,
+      "loss": 6.9228,
+      "step": 22
+    },
+    {
+      "epoch": 0.01038843721770551,
+      "grad_norm": 0.22595034539699554,
+      "learning_rate": 0.00011950903220161285,
+      "loss": 6.9165,
+      "step": 23
+    },
+    {
+      "epoch": 0.01084010840108401,
+      "grad_norm": 0.22315850853919983,
+      "learning_rate": 0.00011305261922200519,
+      "loss": 6.9146,
+      "step": 24
+    },
+    {
+      "epoch": 0.011291779584462511,
+      "grad_norm": 0.2357320338487625,
+      "learning_rate": 0.00010654031292301432,
+      "loss": 6.9185,
+      "step": 25
+    },
+    {
+      "epoch": 0.011291779584462511,
+      "eval_loss": 6.917991638183594,
+      "eval_runtime": 4.0902,
+      "eval_samples_per_second": 911.681,
+      "eval_steps_per_second": 114.174,
+      "step": 25
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 50,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 25,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 1,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1181552492544.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

last-checkpoint/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d0e96fcd4c0713c5b64d98668b44f9bfd98f680338ac8741ce93785709b86c2
+size 6776