|
--- |
|
license: mit |
|
language: |
|
- en |
|
widget: |
|
- text: > |
|
'[CLS]\nQuestion: Are there any pedestrians crossing the road? If yes, how many?\nAnswer: 1\nStudent: One' |
|
example_title: Counting |
|
|
|
tags: |
|
- vision-language |
|
- autonomous-driving |
|
--- |
|
|
|
### What is this? |
|
Lingo-Judge, a novel evaluation metric that aligns closely with human judgment on the LingoQA evaluation suite. |
|
|
|
### How to use |
|
```python |
|
# Import necessary libraries |
|
from transformers import pipeline |
|
|
|
# Define the model name to be used in the pipeline |
|
model_name = 'wayveai/Lingo-Judge' |
|
|
|
# Define the question and its corresponding answer and prediction |
|
question = "Are there any pedestrians crossing the road? If yes, how many?" |
|
answer = "1" |
|
prediction = "Yes, there is one" |
|
|
|
# Initialize the pipeline with the specified model, device, and other parameters |
|
pipe = pipeline("text-classification", model=model_name) |
|
# Format the input string with the question, answer, and prediction |
|
input = f"[CLS]\nQuestion: {question}\nAnswer: {answer}\nStudent: {prediction}" |
|
|
|
# Pass the input through the pipeline to get the result |
|
result = pipe(input) |
|
|
|
# Print the result and score |
|
score = result[0]['score'] |
|
print(score > 0.5, score) |
|
|
|
|
