facebook

/

chameleon-7b

like 164

Follow

AI at Meta 3,292

Hello,

First of all, I would like to thank you for this amazing project. I am evaluating Chameleon's in-context learning ability. However, I think I am missing something about the inference process. When I work with a zero-shot setting, the model outputs are normal. However, with a few-shot setting, the model's responses are awkward. It sometimes avoids answering and occasionally outputs irrelevant characters. I do not encounter this problem in the zero-shot setting. Below you can find the code that I used.

def load_model(self, args) -> None:
    """
    Load the Chameleon model and processor.

    Parameters:
    - args: The arguments to load the model.

    Returns:
    None
    """
    
    from transformers import ChameleonForConditionalGeneration, ChameleonProcessor, BitsAndBytesConfig

    print('Loading Chameleon!!!')

    self.model = ChameleonForConditionalGeneration.from_pretrained(
        args.hf_path,
        device_map="cuda:0",
        torch_dtype=torch.bfloat16,
    ).to(args.device).eval()

    self.processor = ChameleonProcessor.from_pretrained(args.hf_path)
    self.generation_cfg = {
        'do_sample': True,
        'temperature': 0.7,
        'top_p': 0.9,
        'repetition_penalty': 1.2,
    }

    if args.is_zero_cot_active or args.is_few_cot_active:
      self.generation_cfg['max_new_tokens'] = 512
    else:
      self.generation_cfg['max_new_tokens'] = 50

    print('Chameleon loaded!!!')

def calculate_generated_text(self, prompt, vision_x):
    """
    Calculate generated text given a prompt and vision data.

    Parameters:
    - prompt (str): The input prompt.
    - vision_x (list[PIL Images]): List of PIL Images containing vision data.

    Returns:
    Tuple[str, str]: Tuple containing the raw and salt answer text.
    """

   """
   Example Prompt:
   In zero-shot: "<image> <Question> <Options> Answer: "
   In few-shot: "<image> <Question> <Options> Answer: <Answer> <image> <Question> <Options> Answer: "
   """ 
    if self.model is None or self.processor is None:
        raise AttributeError('Model or processor is not initialized. Call load_model first!')

    inputs = self.processor(prompt, images=vision_x, padding=True, return_tensors="pt").to(device=self.model.device, dtype=torch.bfloat16)

    out = self.model.generate(**inputs,  **self.generation_cfg)
    
    generated_text = self.processor.decode(out[0], skip_special_tokens=True)
    
    salt_prompt = prompt.replace("<image>", "")

    salt_answer = generated_text[len(salt_prompt):]

    return generated_text, salt_answer