|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
base_model: t5-small |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- rouge |
|
- accuracy |
|
model-index: |
|
- name: T5_Fine_Tuned_on_Arxiv_Dataset |
|
results: [] |
|
datasets: |
|
- ccdv/arxiv-summarization |
|
language: |
|
- en |
|
--- |
|
|
|
|
|
# T5_Fine_Tuned_on_Arxiv_Dataset |
|
|
|
|
|
## Model Description |
|
This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) designed for summarizing research papers from the Arxiv dataset. It utilizes an abstractive summarization approach to generate concise summaries that capture the main findings and contributions of the papers, facilitating easier understanding of complex academic content. |
|
|
|
## Evaluation |
|
It achieves the following results on the evaluation set: |
|
- Loss: 2.7599 |
|
- Rouge1: 0.1635 |
|
- Rouge2: 0.0548 |
|
- Rougel: 0.1311 |
|
- Rougelsum: 0.1311 |
|
- Generated Length: 18.9852 |
|
|
|
## Model Overview |
|
- **Model Name**: Arxiv Summarization Model |
|
- **Model Type**: Summarization (Abstractive) |
|
- **Version**: 1.0 |
|
- **Date**: [28-Sep-2024] |
|
- **Authors**: Muhammad Ibtisam Afzal |
|
- **Contact Information**: [email protected] |
|
|
|
## Dataset |
|
- **Dataset Name**: ccdv/arxiv-summarization |
|
- **Dataset Description**: This dataset consists of articles from the Arxiv repository, paired with their respective abstracts. It is intended for training and evaluating summarization models in the academic domain. |
|
- **Training/Validation/Test Split**: The dataset was split into training (80%), validation (10%), and test (10%) sets. |
|
- **Data Source**: Hugging Face Datasets Hub |
|
|
|
## Limitations |
|
The model may struggle with highly technical content or specialized jargon that is not well-represented in the training dataset. Additionally, it may produce summaries that lack coherence or completeness for particularly long documents. |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 16 |
|
- eval_batch_size: 16 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 3 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Generated Length | |
|
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:----------------:| |
|
| No log | 1.0 | 305 | 2.8130 | 0.1569 | 0.05 | 0.1256 | 0.1255 | 18.9852 | |
|
| 3.0803 | 2.0 | 610 | 2.7704 | 0.1634 | 0.0546 | 0.1312 | 0.1311 | 18.9852 | |
|
| 3.0803 | 3.0 | 915 | 2.7599 | 0.1635 | 0.0548 | 0.1311 | 0.1311 | 18.9852 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.44.2 |
|
- Pytorch 2.4.1+cu121 |
|
- Datasets 3.0.1 |
|
- Tokenizers 0.19.1 |
|
|
|
## Acknowledgments |
|
Thanks to Hugging Face for providing the infrastructure and datasets necessary for developing and evaluating this model. |
|
|
|
|
