README.md

metadata

library_name: transformers
tags:
  - nli
  - bert
  - natural-language-inference
language:
  - ru
metrics:
  - accuracy
  - f1
  - precision
  - recall
base_model:
  - cointegrated/rubert-tiny2
pipeline_tag: text-classification
model-index:
  - name: rubert-tiny-nli-terra-v0
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: TERRA
          type: NLI
          split: validation
        metrics:
          - type: accuracy
            value: 0.6742671009771987
            name: Accuracy
          - type: f1
            value: 0.6710526315789473
            name: F1
          - type: precision
            value: 0.6754966887417219
            name: Precision
          - type: recall
            value: 0.6666666666666666
            name: Recall

⚠️ Disclaimer: This model is in the early stages of development and may produce low-quality predictions. For better results, consider using the recommended Russian natural language inference models available here.

RuBERT-tiny-nli v1

This model is the second iteration of the RuBERT-tiny2 models for a two-way natural language inference task, utilizing the Russian Textual Entailment Recognition dataset. This model comprises two dense layers in the classifier head to improve inference capabilities. However, it is important to note that the model's performance is currently limited, indicating potential areas for further improvement and fine-tuning.

Usage

How to run the model for NLI:

# !pip install transformers sentencepiece --quiet
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = 'Marwolaeth/rubert-tiny-nli-terra-v1'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
if torch.cuda.is_available():
    model.cuda()

# An example from cointegrated NLI models
premise1 = 'Сократ - человек, а все люди смертны.'
hypothesis1 = 'Сократ никогда не умрёт.'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise1, hypothesis1, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.68763, 'entailment': 0.31237}

# An example concerning sentiments
premise2 = 'Мне не нравятся желтые ковры.'
hypothesis2 = 'Я люблю желтые ковры.'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise2, hypothesis2, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.5894801, 'entailment': 0.41051993}

# A tricky example
# Many NLI models fail to refute premise-hypothesis pairs like:
# 'It is good for our enemies that X' — 'It is good for us that X'
# This contradiction is quite clear, yet many NLI models struggle to accurately identify it, 
# highlighting their limitations in understanding conflicting sentiments in natural language inference.
premise3 = 'Для наших врагов хорошо, что это дерево красное.'
hypothesis3 = 'Для нас хорошо, что это дерево красное.'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise3, hypothesis3, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.54253, 'entailment': 0.45746994}

Model Performance Metrics

The following metrics summarize the performance of the model on the validation dataset:

Metric	Value
Validation Loss	0.6492
Validation Accuracy	67.43%
Validation F1 Score	67.11%
Validation Precision	67.55%
Validation Recall	66.67%
Validation Runtime*	0.2631 seconds
Samples per Second*	1 167.02
Steps per Second*	7.60

*Using T4 GPU with Google Colab