Update README.md

README.md

@@ -32,7 +32,7 @@ pipeline_tag: summarization
 
 # mT5-small based Azerbaijani Summarization
 
-In this project, [Google's Multilingual T5-small](https://github.com/google-research/multilingual-t5) is fine-tuned on [Azerbaijani News Summary Dataset](https://huggingface.co/datasets/nijatzeynalov/azerbaijani-multi-news) for **Summarization** downstream task. The model is trained with 3 epochs, 64 batch size and 10e-4 learning rate. It took almost 12 hours on GPU instance with Ubuntu Server 20.04 LTS image in Microsoft Azure. The max news length is kept as 2048 and max summary length is determined as 128.
+In this model, [Google's Multilingual T5-small](https://github.com/google-research/multilingual-t5) is fine-tuned on [Azerbaijani News Summary Dataset](https://huggingface.co/datasets/nijatzeynalov/azerbaijani-multi-news) for **Summarization** downstream task. The model is trained with 3 epochs, 64 batch size and 10e-4 learning rate. It took almost 12 hours on GPU instance with Ubuntu Server 20.04 LTS image in Microsoft Azure. The max news length is kept as 2048 and max summary length is determined as 128.
 
 
 mT5 is a multilingual variant of __T5__ and only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual)
@@ -64,4 +64,57 @@ The T5 model, pre-trained on C4, achieves state-of-the-art results on many NLP b
 
 mT5 is pre-trained only by unsupervised manner with multiple languages, and it’s not trained for specific downstream tasks. To dare say, this pre-trained model has ability to build correct text in Azerbaijani, but it doesn’t have any ability for specific tasks, such as, summarization, correction, machine translation, etc.
 
-Therefore I trained (fine-tune) this model for summarization in Azerbaijani using [Azerbaijani News Summary Dataset](https://huggingface.co/datasets/nijatzeynalov/azerbaijani-multi-news).
+In HuggingFace, several sizes of mT5 models are available, and here I used small one (google/mt5-small). Therefore I trained (fine-tune) this model for summarization in Azerbaijani using [Azerbaijani News Summary Dataset](https://huggingface.co/datasets/nijatzeynalov/azerbaijani-multi-news).
+
+
+## Training hyperparameters
+
+__mT5-based-azerbaijani-summarize__ model training took almost 12 hours on GPU instance with Ubuntu Server 20.04 LTS image in Microsoft Azure. The following hyperparameters were used during training:
+
+- learning_rate: 0.0005
+- train_batch_size: 2
+- eval_batch_size: 1
+- seed: 42
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 64
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 90
+- num_epochs: 10
+
+## Dataset
+
+Model was trained on [__az-news-summary__ dataset](https://huggingface.co/datasets/nijatzeynalov/azerbaijani-multi-news), a comprehensive and diverse dataset comprising 143k (143,448) Azerbaijani news articles extracted using a set of carefully designed heuristics. 
+ 
+The dataset covers common topics for news reports include war, government, politics, education, health, the environment, economy, business, fashion, entertainment, and sport, as well as quirky or unusual events.
+
+This dataset has 3 splits: _train_, _validation_, and _test_. \
+Token counts are white space based.
+
+| Dataset Split | Number of Instances |     Size (MB)         |
+| ------------- | --------------------|:----------------------|
+| Train         | 100,413             |      150              |
+| Validation    | 14,344              |      21.3             |
+| Test          | 28,691              |      42.8             |
+
+
+## Training results with comparison
+
+It achieves the following results on the test set:
+
+Rouge1: 39.4222
+Rouge2: 24.8624
+Rougel: 32.2487
+
+For __Azerbaijani text summarization downstream task__, mT5-multilingual-XLSum has also been developed on the 45 languages of [XL-Sum](https://huggingface.co/datasets/csebuetnlp/xlsum) dataset. For finetuning details and scripts,
+see the [paper](https://aclanthology.org/2021.findings-acl.413/) and the [official repository](https://github.com/csebuetnlp/xl-sum). .
+
+Scores on the XL-Sum test sets for Azerbaijani are as follows:
+
+Rouge1: 21.4227
+Rouge2: 9.5214
+Rougel: 19.3331
+
+As seen from the numbers, __mT5-based-azerbaijani-summarize__ model achieves dramatically better performance than __mT5_multilingual_XLSum__.
+
+## Using this model in transformers