AIDC-AI

/

Ovis1.6-Gemma2-27B

like 57

Follow

AIDC-AI 292

@himasai9711 Of course there's a way to get the confidence score~ Since the 'compute_transition_scores' method is encapsulated in the LLM part of the model, you can get score by

outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)  
transition_scores = model.llm.compute_transition_scores(outputs.sequences, outputs.scores, normalize_logits=True) 

@chenyh7 Can you let me know how we can do this for Ovis models ? my inference code for gemma is

with torch.inference_mode():
    gen_kwargs = dict(
        max_new_tokens=1024,
        do_sample=False,
        top_p=None,
        top_k=None,
        temperature=None,
        repetition_penalty=None,
        eos_token_id=model.generation_config.eos_token_id,
        pad_token_id=text_tokenizer.pad_token_id,
        use_cache=True
    )
     output_ids = model.generate(input_ids, pixel_values=pixel_values, attention_mask=attention_mask, **gen_kwargs)[0]
    output = text_tokenizer.decode(output_ids, skip_special_tokens=True)

The above code generate my output how can i incorporate the confidence score here

@chenyh7 Can you let me know how we can do this for Ovis models ? my inference code for gemma is

with torch.inference_mode():
    gen_kwargs = dict(
        max_new_tokens=1024,
        do_sample=False,
        top_p=None,
        top_k=None,
        temperature=None,
        repetition_penalty=None,
        eos_token_id=model.generation_config.eos_token_id,
        pad_token_id=text_tokenizer.pad_token_id,
        use_cache=True
    )
     output_ids = model.generate(input_ids, pixel_values=pixel_values, attention_mask=attention_mask, **gen_kwargs)[0]
    output = text_tokenizer.decode(output_ids, skip_special_tokens=True)

The above code generate my output how can i incorporate the confidence score here

@himasai9711 Here‘s the example:

with torch.inference_mode():
    gen_kwargs = dict(
        max_new_tokens=1024,
        do_sample=False,
        top_p=None,
        top_k=None,
        temperature=None,
        repetition_penalty=None,
        eos_token_id=model.generation_config.eos_token_id,
        pad_token_id=text_tokenizer.pad_token_id,
        use_cache=True,
        return_dict_in_generate=True,
        output_scores=True
    )
    output_dict = model.generate(input_ids, pixel_values=pixel_values, attention_mask=attention_mask, **gen_kwargs)
    output = text_tokenizer.decode(outputs[0][0], skip_special_tokens=True)
    print(f'Output:\n{output}')
    transition_scores = model.llm.compute_transition_scores(output_dict.sequences, output_dict.scores, normalize_logits=True) 
    print(transition_scores.cpu().numpy())

@chenyh7 thanks for giving the sample , I have a small doubt as before I'm using this for document extraction task from pdf to json, let say for example i have a text in the image is Apple and model predicted it as Aple so the confident_scores retuned is [99.99,99.99,99.99,99.99] but the confidence score of missing token is missing it will effect my overall confidence of the word right, how can i solve that?

As this is extraction problem i dont have any ground truth..