ColonGPT-v1 / README.md
DanielJi's picture
Add pipeline tag (#1)
7b0361c verified
metadata
license: apache-2.0
datasets:
  - ai4colonoscopy/ColonINST-v1
language:
  - en
metrics:
  - accuracy
base_model:
  - microsoft/phi-1_5
library_name: adapter-transformers
pipeline_tag: image-text-to-text
tags:
  - medical
  - colonoscopy
  - polyp

ColonGPT (A colonoscopy-specific multimodal Language Model)


The Gradio Web UI allows you to use our examples or upload your images for inference.

๐Ÿ“– Paper | ๐Ÿ  Home

This is the merged weights of ColonGPT-v1-phi1.5-siglip-lora, including vision encoder (siglip) + language model (phi-1.5), and other fine-tuned weights on our ColonINST.

Our ColonGPT is a standard multimodal language model, which contains four basic components: a language tokenizer, an visual encoder (๐Ÿค— SigLIP-SO), a multimodal connector, and a language model (๐Ÿค— Phi1.5). In this huggingface page, we provide a quick start for convenient of new users. For further details about ColonGPT, we highly recommend visiting our homepage. There, you'll find comprehensive usage instructions for our model and the latest advancements in intelligent colonoscopy technology.

Quick start

Here is a code snippet to show you how to quickly try-on our ColonGPT model with transformers. For convenience, we manually combined some configuration and code files and merged the weights. Please note that this is a quick code, we recommend you installing ColonGPT's source code to explore more.

  • Before running the snippet, you only need to install the following minimium dependencies.

    conda create -n quickstart python=3.10
    conda activate quickstart
    pip install torch transformers accelerate pillow
    
  • Then you can use python script/quick_start/quickstart.py to start.

    import torch
    import transformers
    from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria
    from PIL import Image
    import warnings
    
    transformers.logging.set_verbosity_error()
    transformers.logging.disable_progress_bar()
    warnings.filterwarnings('ignore')
    
    device = 'cuda'  # or cpu
    torch.set_default_device(device)
    
    model_name = "ai4colonoscopy/ColonGPT-v1"
    
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float16,  # or float32 for cpu
        device_map='auto',
        trust_remote_code=True
    )
    
    tokenizer = AutoTokenizer.from_pretrained(
        model_name,
        trust_remote_code=True
    )
    
    class KeywordsStoppingCriteria(StoppingCriteria):
        def __init__(self, keyword, tokenizer, input_ids):
            self.keyword_id = tokenizer(keyword).input_ids
            self.tokenizer = tokenizer
            self.start_len = input_ids.shape[1]
    
        def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
            for keyword_id in self.keyword_id:
                if keyword_id in input_ids[0, -len(self.keyword_id):]:
                    return True
            return False
    
    prompt = "Describe what you see in the image." 
    text = f"USER: <image>\n{prompt} ASSISTANT:"
    text_chunks = [tokenizer(chunk).input_ids for chunk in text.split('<image>')]
    input_ids = torch.tensor(text_chunks[0] + [-200] + text_chunks[1], dtype=torch.long).unsqueeze(0).to(device)
    
    image = Image.open('cache/examples/example2.png')
    image_tensor = model.process_images([image], model.config).to(dtype=model.dtype, device=device)
    
    stop_str = "<|endoftext|>"
    stopping_criteria = KeywordsStoppingCriteria(stop_str, tokenizer, input_ids)
    
    output_ids = model.generate(
        input_ids,
        images=image_tensor,
        do_sample=False,
        temperature=0,
        max_new_tokens=512,
        use_cache=True,
        stopping_criteria=[stopping_criteria]
    )
    
    outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:]).replace("<|endoftext|>", "").strip()
    print(outputs)
    

License

This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses. The content of this project itself is licensed under the Apache license 2.0.