T5_Fine_Tuned_on_Arxiv_Dataset

Model Description

This model is a fine-tuned version of t5-small designed for summarizing research papers from the Arxiv dataset. It utilizes an abstractive summarization approach to generate concise summaries that capture the main findings and contributions of the papers, facilitating easier understanding of complex academic content.

Evaluation

It achieves the following results on the evaluation set:

• Loss: 2.7599
• Rouge1: 0.1635
• Rouge2: 0.0548
• Rougel: 0.1311
• Rougelsum: 0.1311
• Generated Length: 18.9852

Model Overview

• Model Name: Arxiv Summarization Model
• Model Type: Summarization (Abstractive)
• Version: 1.0
• Date: [28-Sep-2024]
• Authors: Muhammad Ibtisam Afzal
• Contact Information: [email protected]

Dataset

• Dataset Name: ccdv/arxiv-summarization
• Dataset Description: This dataset consists of articles from the Arxiv repository, paired with their respective abstracts. It is intended for training and evaluating summarization models in the academic domain.
• Training/Validation/Test Split: The dataset was split into training (80%), validation (10%), and test (10%) sets.
• Data Source: Hugging Face Datasets Hub

Limitations

The model may struggle with highly technical content or specialized jargon that is not well-represented in the training dataset. Additionally, it may produce summaries that lack coherence or completeness for particularly long documents.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Generated Length
No log	1.0	305	2.8130	0.1569	0.05	0.1256	0.1255	18.9852
3.0803	2.0	610	2.7704	0.1634	0.0546	0.1312	0.1311	18.9852
3.0803	3.0	915	2.7599	0.1635	0.0548	0.1311	0.1311	18.9852

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.19.1

Acknowledgments

Thanks to Hugging Face for providing the infrastructure and datasets necessary for developing and evaluating this model.