paligemma-3b-mix-448-ft-TableDetection

This model is a mixed precision fine-tuned version of google/paligemma-3b-mix-448 on ucsahin/pubtables-detection-1500-samples dataset. It achieves the following results on the evaluation set:

• Loss: 1.3544

Model Details

• This model is a multimodal language model fine-tuned for the task of detecting tables in images given textual prompts. The model utilizes a combination of image and text inputs to predict bounding boxes around tables within the provided images.
• The primary purpose of this model is to assist in automating the process of table detection within images. It can be utilized in various applications such as document processing, data extraction, and image analysis, where identifying tables within images is essential.

Inputs:

• Image: The model requires an image containing one or more tables as input. The image should be in a standard format such as JPEG or PNG.
• Text Prompt: Additionally, a text prompt is required to guide the model's attention towards the task of table detection. The prompt should clearly indicate the desired action. Please use "detect table" as your text prompt.

Outputs:

• Bounding Boxes: The model outputs the location for the bounding box coordinates in the form of special <loc[value]> tokens, where value is a number that represents a normalized coordinate. Each detection is represented by four location coordinates in the order y_min, x_min, y_max, x_max, followed by the label that was detected in that box. To convert values to coordinates, you first need to divide the numbers by 1024, then multiply y by the image height and x by its width. This will give you the coordinates of the bounding boxes, relative to the original image size. If everything goes smoothly, the model will output a text similar to "<loc[value]><loc[value]><loc[value]><loc[value]> table; <loc[value]><loc[value]><loc[value]><loc[value]> table" depending on the number of tables detected in the image. Then, you can use the following script to convert the text output into PASCAL VOC formatted bounding boxes.

import re

def post_process(bbox_text, image_width, image_height):
    loc_values_str = [bbox.strip() for bbox in bbox_text.split(";")]
    
    converted_bboxes = []
    for loc_value_str in loc_values_str:
        loc_values = re.findall(r'<loc(\d+)>', loc_value_str)
        loc_values = [int(x) for x in loc_values]
        loc_values = loc_values[:4] 
        
        loc_values = [value/1024 for value in loc_values]
        # convert to (xmin, ymin, xmax, ymax)
        loc_values = [
            int(loc_values[1]*image_width), int(loc_values[0]*image_height), 
            int(loc_values[3]*image_width), int(loc_values[2]*image_height), 
        ]
        converted_bboxes.append(loc_values)
    
    return converted_bboxes

How to Get Started with the Model

In Transformers, you can load the model as follows:

from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
import torch

model_id = "ucsahin/paligemma-3b-mix-448-ft-TableDetection"

device = "cuda:0"
dtype = torch.bfloat16

model = PaliGemmaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=dtype,
    device_map=device
)

processor = PaliGemmaProcessor.from_pretrained(model_id)

For inference, you can use the following:

# # Instruct the model to detect tables
prompt = "detect table"
model_inputs = processor(text=prompt, images=image, return_tensors="pt").to(device)
input_len = model_inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**model_inputs, max_new_tokens=128, do_sample=False)
    generation = generation[0][input_len:]
    bbox_text = processor.decode(generation, skip_special_tokens=True)
    print(bbox_text)

Warning: You can also load a quantized 4-bit or 8-bit model using bitsandbytes. Beware though that the model can generate outputs that can require further post-processing for example five locations tags "<loc[value]>" instead of four, and different labels other than "table". The provided post-processing script should handle the first case.

Use the following to load the 4-bit quantized model:

from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor, BitsAndBytesConfig
import torch

model_id = "ucsahin/paligemma-3b-mix-448-ft-TableDetection"

device = "cuda:0"
dtype = torch.bfloat16

bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=dtype
)

model = PaliGemmaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=dtype,
    device_map=device,
    quantization_config=bnb_config
)

processor = PaliGemmaProcessor.from_pretrained(model_id)

Bias, Risks, and Limitations

Please refer to google/paligemma-3b-mix-448 for bias, risks and limitations.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• bf16: True mixed precision
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
2.957	0.1775	15	2.1300
1.9656	0.3550	30	1.8421
1.6716	0.5325	45	1.6898
1.5514	0.7101	60	1.5803
1.5851	0.8876	75	1.5271
1.4134	1.0651	90	1.4771
1.3566	1.2426	105	1.4528
1.3093	1.4201	120	1.4227
1.2897	1.5976	135	1.4115
1.256	1.7751	150	1.4007
1.2666	1.9527	165	1.3678
1.2213	2.1302	180	1.3744
1.0999	2.3077	195	1.3633
1.1931	2.4852	210	1.3606
1.0722	2.6627	225	1.3619
1.1485	2.8402	240	1.3544

Framework versions

• PEFT 0.11.1
• Transformers 4.42.0.dev0
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1