nm-testing

/

NVLM-D-72B-FP8-dynamic

like 3

Follow

NM Testing 50

@imgautamI9 Here's an example: https://github.com/vllm-project/vllm/blob/bf21481ddef2fa9bb96c13ba1f80072abdae3eb7/examples/offline_inference/vision_language.py#L366

Please note that this checkpoint in particular has been uploaded by nm-testing and is therefore not guaranteed to produce good recovery.

For the FP8 dynamic Model it is throwing this error : " RuntimeError: Input type (c10::BFloat16) and bias type (c10::Half) should be the same " while i'm using the same code for test inference.