|
--- |
|
library_name: transformers |
|
tags: |
|
- cross-encoder |
|
- sentence-transformers |
|
datasets: |
|
- phonemetransformers/CHILDES |
|
language: |
|
- en |
|
base_model: |
|
- sentence-transformers/all-mpnet-base-v2 |
|
pipeline_tag: text-classification |
|
--- |
|
|
|
# Model Card for Coherence Testing Model |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
This model is a fine-tuned version of the `sentence-transformers/all-mpnet-base-v2` designed specifically for coherence testing in dialogues. Leveraging the cross-encoder architecture from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) library, it is intended to evaluate the relevance and coherence of responses given a prompt or question. |
|
|
|
- **Developed by:** Enoch Levandovsky |
|
- **Model type:** Cross-encoder |
|
- **Language(s):** English |
|
- **License:** Check the repository for more information |
|
- **Finetuned from model:** sentence-transformers/all-mpnet-base-v2 |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [Model on Hugging Face](https://huggingface.co/enochlev/coherence-all-mpnet-base-v2) |
|
- **Space Demo:** [Coherence Testing Space](https://huggingface.co/spaces/enochlev/coherence-all-mpnet-base-v2-space) |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
This model is designed to evaluate the coherence of a response to a given question or prompt. It can be directly used to enhance chatbots or dialogue systems by predicting how coherent or relevant a response is, thus improving the quality of conversational agents. |
|
|
|
### Downstream Use |
|
|
|
This model can be fine-tuned further for specific dialogue systems or used as a component in larger conversational AI frameworks to ensure responses are meaningful and contextually appropriate. |
|
|
|
### Out-of-Scope Use |
|
|
|
This model is not intended for applications requiring complex sentiment analysis, emotional tone recognition, or tasks outside dialogue coherence assessment. |
|
|
|
## How to Get Started with the Model |
|
|
|
You can use the model as follows: |
|
|
|
```python |
|
from sentence_transformers import CrossEncoder |
|
model = CrossEncoder('enochlev/coherence-all-mpnet-base-v2') |
|
output = model.predict([["What is your favorite color?", "Blue!"], |
|
["Do you like playing outside?", "I like ice cream."], |
|
["What is your favorite animal?", "I like dogs!"], |
|
["Do you want to go to the park?", "Yes, I want to go on the swings!"], |
|
["What is your favorite food?", "I like playing with blocks."], |
|
["Do you have a pet?", "Yes, I have a cat named Whiskers."], |
|
["What is your favorite thing to do on a sunny day?", "I like playing soccer with my friends."]]) |
|
|
|
print(output) |
|
``` |
|
|
|
The output array represents coherence scores where higher scores indicate greater coherence. |
|
|
|
## Results |
|
|
|
Example outputs reflect coherent or relevant responses with scores closer to 1. For instance: |
|
|
|
```plaintext |
|
Output >>> array([0.88097143, 0.04521223, 0.943173 , 0.9436357 , 0.04369843, |
|
0.94450355, 0.8392763 ], dtype=float32) |
|
``` |
|
|
|
## Evaluation & Limitations |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
The model has been fine-tuned and evaluated using the CHILDES dataset to ensure it captures conversational coherence effectively. |
|
|
|
### Recommendations |
|
|
|
Users should be aware that while the model predicts coherence, it may not fully capture nuanced conversational elements such as sarcasm or humor. |
|
|
|
## Environmental Impact |
|
|
|
Please refer to the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) for estimating carbon emissions. Details specific to training this model are not available but consider general best practices to minimize environmental impact. |
|
|
|
## Citation |
|
|
|
To cite this model, please provide appropriate credit to the Hugging Face repository page and the original model creator, Enoch Levandovsky. |
