Hallucinations & 8/16-bit build

#10

by roblek - opened 10 days ago

roblek

10 days ago

This 4-bit ONNX version of the model seems to hallucinate frequently:

Are there any specific GeneratorParams.search_options you would recommend to reduce the hallucinations?
What is the procedure for building an 8-bit or 16-bit ONNX version of the model?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment