library_name: transformers
license: apache-2.0
base_model: t5-small
tags:
- generated_from_trainer
metrics:
- rouge
- accuracy
model-index:
- name: T5_Fine_Tuned_on_Arxiv_Dataset
results: []
datasets:
- ccdv/arxiv-summarization
language:
- en
T5_Fine_Tuned_on_Arxiv_Dataset
Model Description
This model is a fine-tuned version of t5-small designed for summarizing research papers from the Arxiv dataset. It utilizes an abstractive summarization approach to generate concise summaries that capture the main findings and contributions of the papers, facilitating easier understanding of complex academic content.
Evaluation
It achieves the following results on the evaluation set:
- Loss: 2.7599
- Rouge1: 0.1635
- Rouge2: 0.0548
- Rougel: 0.1311
- Rougelsum: 0.1311
- Generated Length: 18.9852
Model Overview
- Model Name: Arxiv Summarization Model
- Model Type: Summarization (Abstractive)
- Version: 1.0
- Date: [28-Sep-2024]
- Authors: Muhammad Ibtisam Afzal
- Contact Information: [email protected]
Dataset
- Dataset Name: ccdv/arxiv-summarization
- Dataset Description: This dataset consists of articles from the Arxiv repository, paired with their respective abstracts. It is intended for training and evaluating summarization models in the academic domain.
- Training/Validation/Test Split: The dataset was split into training (80%), validation (10%), and test (10%) sets.
- Data Source: Hugging Face Datasets Hub
Limitations
The model may struggle with highly technical content or specialized jargon that is not well-represented in the training dataset. Additionally, it may produce summaries that lack coherence or completeness for particularly long documents.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Generated Length |
|---|---|---|---|---|---|---|---|---|
| No log | 1.0 | 305 | 2.8130 | 0.1569 | 0.05 | 0.1256 | 0.1255 | 18.9852 |
| 3.0803 | 2.0 | 610 | 2.7704 | 0.1634 | 0.0546 | 0.1312 | 0.1311 | 18.9852 |
| 3.0803 | 3.0 | 915 | 2.7599 | 0.1635 | 0.0548 | 0.1311 | 0.1311 | 18.9852 |
Framework versions
- Transformers 4.44.2
- Pytorch 2.4.1+cu121
- Datasets 3.0.1
- Tokenizers 0.19.1
Acknowledgments
Thanks to Hugging Face for providing the infrastructure and datasets necessary for developing and evaluating this model.