Hugging Face's logo

Spaces:
edugp
/
perplexity-lenses
Runtime error

App Files Community
1
perplexity-lenses / perplexity_lenses /perplexity.py

Commit History

Run tokenizer before computing perplexity and format
7b62017

edugp commited on

Replicate default cc_net preprocessing at inference time on KenlmModel.get_perplexity
0def03f

edugp commited on

Check if file exists before attempting to remove
ab7449f

edugp commited on

Remove also the '{language}.sp.model' file on failure
38b6530

edugp commited on

Remove corrupt KenLM model files.
9ec7b19

edugp commited on

Support visualizing both sentences and whole documents. Smooth down color assignment in visualization.
a86046b

edugp commited on