Upload folder using huggingface_hub

README.md

@@ -1,3 +1,206 @@
----
-license: apache-2.0
----
+# Azerbaijani Named Entity Recognition (NER) with XLM-RoBERTa Large
+
+**Repository on Hugging Face**: [IsmatS/xlm_roberta_large_az_ner](https://huggingface.co/IsmatS/xlm_roberta_large_az_ner)  
+**Repository on GitHub**: [Named Entity Recognition](https://github.com/Ismat-Samadov/Named_Entity_Recognition)
+
+## Project Overview
+
+This project leverages `xlm-roberta-large`, a multilingual transformer model, fine-tuned for Azerbaijani Named Entity Recognition (NER). The model identifies various named entities, including persons, organizations, dates, etc., using a dataset specially designed for the Azerbaijani language.
+
+## Table of Contents
+
+1. [Setup and Dependencies](#setup-and-dependencies)
+2. [Dataset](#dataset)
+3. [Model Architecture](#model-architecture)
+4. [Training Process](#training-process)
+5. [Training Metrics and Results](#training-metrics-and-results)
+6. [Evaluation and Detailed Metrics Explanation](#evaluation-and-detailed-metrics-explanation)
+7. [Saving and Loading the Model](#saving-and-loading-the-model)
+8. [Example Inference](#example-inference)
+9. [Troubleshooting and Notes](#troubleshooting-and-notes)
+
+---
+
+## Setup and Dependencies
+
+Install the required libraries:
+
+```python
+!pip install transformers datasets seqeval huggingface_hub
+```
+
+### Imports
+
+The project requires `transformers`, `datasets`, `torch`, and `seqeval` libraries.
+
+## Dataset
+
+The Azerbaijani NER dataset includes entities such as **PERSON**, **LOCATION**, **ORGANISATION**, etc., and is hosted on Hugging Face.
+
+```python
+from datasets import load_dataset
+dataset = load_dataset("LocalDoc/azerbaijani-ner-dataset")
+```
+
+A preprocessing function ensures token and NER tag formatting:
+
+```python
+def preprocess_example(example):
+    example["tokens"] = ast.literal_eval(example["tokens"])
+    example["ner_tags"] = list(map(int, ast.literal_eval(example["ner_tags"])))
+    return example
+
+dataset = dataset.map(preprocess_example)
+```
+
+## Model Architecture
+
+The model is based on `xlm-roberta-large`, designed to handle multilingual text processing.
+
+### Tokenization and Label Alignment
+
+A custom function `tokenize_and_align_labels` tokenizes input while aligning entity labels with tokens.
+
+```python
+tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
+
+def tokenize_and_align_labels(example):
+    tokenized_inputs = tokenizer(
+        example["tokens"], truncation=True, is_split_into_words=True,
+        padding="max_length", max_length=128
+    )
+    # Alignment code follows here...
+    return tokenized_inputs
+```
+
+---
+
+## Training Process
+
+Training uses the Hugging Face `Trainer`, which handles the training loop, metrics computation, and model checkpointing.
+
+### Model Initialization
+
+```python
+from transformers import AutoModelForTokenClassification
+model = AutoModelForTokenClassification.from_pretrained(
+    "xlm-roberta-large", num_labels=len(label_list)
+)
+```
+
+### Define Evaluation Metrics
+
+The following metrics help evaluate the model’s accuracy and performance in recognizing and classifying entities.
+
+```python
+from seqeval.metrics import precision_score, recall_score, f1_score, classification_report
+
+def compute_metrics(p):
+    # Metric computation code
+    return {
+        "precision": precision_score(true_labels, true_predictions),
+        "recall": recall_score(true_labels, true_predictions),
+        "f1": f1_score(true_labels, true_predictions),
+    }
+```
+
+### Training Configuration
+
+```python
+training_args = TrainingArguments(
+    output_dir="./results", evaluation_strategy="epoch", save_strategy="epoch",
+    learning_rate=2e-5, per_device_train_batch_size=128, per_device_eval_batch_size=128,
+    num_train_epochs=12, weight_decay=0.005, fp16=True, logging_dir='./logs',
+    save_total_limit=2, load_best_model_at_end=True, metric_for_best_model="f1", report_to="none"
+)
+```
+
+---
+
+## Training Metrics and Results
+
+During training, metrics for each epoch were recorded. These metrics include `Training Loss`, `Validation Loss`, `Precision`, `Recall`, and `F1-Score` for both training and validation sets.
+
+| Epoch | Training Loss | Validation Loss | Precision | Recall | F1-Score |
+|-------|---------------|-----------------|-----------|--------|----------|
+| 1     | 0.4075        | 0.2538          | 0.7689    | 0.7214 | 0.7444   |
+| 2     | 0.2556        | 0.2497          | 0.7835    | 0.7245 | 0.7528   |
+| 3     | 0.2144        | 0.2488          | 0.7509    | 0.7489 | 0.7499   |
+| 4     | 0.1934        | 0.2571          | 0.7686    | 0.7404 | 0.7542   |
+| 5     | 0.1698        | 0.2757          | 0.7458    | 0.7537 | 0.7497   |
+| 6     | 0.1526        | 0.2881          | 0.7831    | 0.7284 | 0.7548   |
+| 7     | 0.1443        | 0.3034          | 0.7585    | 0.7381 | 0.7481   |
+| 8     | 0.1268        | 0.3113          | 0.7456    | 0.7509 | 0.7482   |
+| 9     | 0.1194        | 0.3316          | 0.7393    | 0.7495 | 0.7444   |
+| 10    | 0.1094        | 0.3448          | 0.7543    | 0.7372 | 0.7456   |
+| 11    | 0.1029        | 0.3549          | 0.7519    | 0.7413 | 0.7466   |
+
+These metrics demonstrate the model’s performance progression through each epoch, highlighting how it optimizes towards lower losses and higher precision, recall, and F1-scores.
+
+---
+
+## Evaluation and Detailed Metrics Explanation
+
+After training, the model was evaluated across various entity types with the following results:
+
+| Entity       | Precision | Recall | F1-score | Support |
+|--------------|-----------|--------|----------|---------|
+| ART          | 0.41      | 0.19   | 0.26     | 1828    |
+| DATE         | 0.53      | 0.49   | 0.51     | 834     |
+| EVENT        | 0.67      | 0.51   | 0.58     | 63      |
+| FACILITY     | 0.74      | 0.68   | 0.71     | 1134    |
+| LAW          | 0.62      | 0.58   | 0.60     | 1066    |
+| LOCATION     | 0.81      | 0.79   | 0.80     | 8795    |
+| MONEY        | 0.59      | 0.56   | 0.58     | 555     |
+| ORGANISATION | 0.70      | 0.69   | 0.70     | 554     |
+| PERCENTAGE   | 0.80      | 0.82   | 0.81     | 3502    |
+| PERSON       | 0.90      | 0.82   | 0.86     | 7007    |
+| PRODUCT      | 0.83      | 0.84   | 0.84     | 2624    |
+| TIME         | 0.60      | 0.53   | 0.57     | 1584    |
+
+### Explanation of Metrics
+
+- **Precision**: Represents the proportion of correctly identified entities out of all entities predicted by the model. High precision is vital in NER tasks to reduce false positives, ensuring only actual entities are classified.
+- **Recall**: Indicates the proportion of correctly identified entities out of all actual entities present in the dataset. Higher recall captures all relevant entities, which is essential to avoid missing important information.
+- **F1-Score**: The harmonic mean of precision and recall, balancing both metrics. A high F1-score suggests an effective trade-off between precision and recall, ideal for NER where both metrics are crucial for accurate entity recognition.
+
+## Saving and Loading the Model
+
+Save the model and tokenizer for future use or further fine-tuning:
+
+```python
+save_directory = "./xlm-roberta-large"
+model.save_pretrained(save_directory)
+tokenizer.save_pretrained(save_directory)
+```
+
+---
+
+## Example Inference
+
+Load the saved model for inference, utilizing the Hugging Face pipeline for NER.
+
+```python
+from transformers import pipeline
+
+tokenizer = AutoTokenizer.from_pretrained(save_directory)
+model = AutoModelForTokenClassification.from_pretrained(save_directory)
+device = 0 if torch.cuda.is_available() else -1
+nlp_ner = pipeline("ner", model=model, tokenizer=tokenizer
+
+, aggregation_strategy="simple", device=device)
+
+# Example sentence
+test_texts = ["Shahla Khuduyeva və Pasha Sığorta şirkəti haqqında məlumat."]
+evaluate_model(test_texts, [["B-PERSON", "B-ORGANISATION"]])
+```
+
+---
+
+## Troubleshooting and Notes
+
+- **Token Alignment**: Tokenization and label alignment must be carefully handled, especially when dealing with subwords.
+- **Memory Management**: Adjust batch size if encountering GPU memory limitations.
+- **Early Stopping**: Configured with a patience of 5 epochs to avoid overfitting, automatically halting training if validation performance plateaus.
+
+This README provides detailed information on the project setup, training, evaluation, and inference for the `xlm-roberta-large` NER model fine-tuned for Azerbaijani text. The model's performance on various entity types and the clear explanation of metrics make it a comprehensive resource for Azerbaijani NER tasks.