Update README.md

README.md

@@ -56,18 +56,25 @@ Fine tuning is done using the `train` split of the GLUE MNLI dataset and the per
 
 `validation_mismatched` means validation examples are not derived from the same sources as those in the training set and therefore not closely resembling any of the examples seen at training time.
 
+Data splits for the mnli dataset are the following
+|train |validation_matched|validation_mismatched|
+|-----:|-----------------:|--------------------:|
+|392702|              9815|                 9832|
 ## Fine-tuning procedure
 Fine tuned on a Graphcore IPU-POD64 using `popxl`.     
 
 Prompt sentences are tokenized and packed together to form 1024 token sequences, following [HF packing algorithm](https://github.com/huggingface/transformers/blob/v4.20.1/examples/pytorch/language-modeling/run_clm.py). No padding is used.
+The packing process works in groups of 1000 examples and discards any remainder from each group that isn't a whole sequence.
+For the 392,702 training examples this gives a total of 17,762 sequences per epoch. 
+
 Since the model is trained to predict the next token, labels are simply the input sequence shifted by one token.
 Given the training format, no extra care is needed to account for different sequences: the model does not need to know which sentence a token belongs to.
 
 ### Hyperparameters:
-- epochs: 
 - optimiser: AdamW (beta1: 0.9, beta2: 0.999, eps: 1e-6, weight decay: 0.0, learning rate: 5e-6)
 - learning rate schedule: warmup schedule (min: 1e-7, max: 5e-6, warmup proportion: 0.005995)
 - batch size: 128
+- training steps: 300. Each epoch consists of ceil(17,762/128) steps, hence 300 steps are approximately 2 epochs.  
 
 ## Performance
 The resulting model matches SOTA performance with 82.5% accuracy.