Model Details
The CFT-CLIP was developed by HUMANE Lab researchers at Soongsil University to understand news thumbnail representativeness by counterfactual text-guided contrastive language-image pretraining.
Model Date
January 2024
Model Type
The model uses a ViT-L/14 transformer architecture as an image encoder and a causal text transformer as a text encoder. These encoders initialized weight for openai/clip-vit-large-patch14 before training. It is trained that the similarity of positive (image, text) pairs is high, and the similarity of in-batch negatives and hard negatives is low via contrastive loss.
Input: image and text
output: image and text representation
Intended Use
The model is intended as a research output for research communities.
Primary intended uses
The primary intended users of these models are AI researchers.
Out-of-Scope Use Cases
The model was not intentionally trained or evaluated in any language other than English. Therefore, use of the model should be limited to English use cases.
Factors
Environment
This model was trained on a machine equipped with AMD Ryzen Threadripper Pro 5975WX CPU, three Nvidia RTX A6000 GPUs (48GB per GPU), and 256GB RAM. The experiments were conducted on Python 3.9, Pytorch 1.10.1, Transformers 4.29.2, LAVIS 1.0.2, and SentenceTransformer 2.2.2. Five random seeds were used for repeated experiments: 0, 1, 2, 3, and 4. The temperature used for adjusting the masked token prediction is set as 2.0.
Card Prompts
Relevant factors
We trained the models with the AdamW optimizer with the initial learning rate of 1e-4, updated by the cosine annealing scheduler.The minibatch size is 128. The temperature τ in the loss equation is 0.05. Other hyperparameters were optimized by random search using a validation set. Model training was early-stopped when the validation loss was not decreased five times consecutively, measured for every 20 iterations.
Evaluation factors
Metrics
Model performance measures: F1-score between model predictions and labels and Spearman between cosine similarity of models between labels.
Decision thresholds: validation
Approaches to uncertainty and variability: Measure by changing the random seed 5 times
Data
Training Data
The model was trained using the summary text and thumbnail image for the image in the first paragraph of the publicly available BBC English Dataset. The original implementation had two variants: one using a NELA-GT-2021 and the other using the titles instead of summary text from BBC Dataset.
Evaluation Data
In NELA-GT-2021, annotation was performed by randomly sampling 1,000 in 10,000 samples not included in the train and valid set. Ethical Considerations Because CLIP's weights are used, potential bias in CLIP and potential bias in the data used for learning may also be included.