metadata

title: Perplexity
emoji: 🌖
colorFrom: purple
colorTo: pink
sdk: gradio
sdk_version: 4.7.1
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - evaluate
  - metric
description: >-
  Perplexity metric implemented by d-Matrix. Perplexity (PPL) is one of the most
  common metrics for evaluating language models. It is defined as the
  exponentiated average negative log-likelihood of a sequence, calculated with
  exponent base `e`. For more information, see
  https://huggingface.co/docs/transformers/perplexity

Metric Card for Perplexity

Metric Description

Perplexity metric implemented by d-Matrix. Perplexity (PPL) is one of the most common metrics for evaluating language models. It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base e. For more information, see https://huggingface.co/docs/transformers/perplexity

How to Use

At minimum, this metric requires the model and text as inputs.

>>> perplexity = evaluate.load("d-matrix/perplexity", module_type="metric")
>>> input_texts = ["lorem ipsum", "Happy Birthday!", "Bienvenue"]
>>> results = perplexity.compute(model='distilgpt2',text=input_texts)
>>> print(results)
{'accuracy': 1.0}