zer0int

/

CLIP-GmP-ViT-L-14

like 349

Of course! It has an MIT license, same as the original CLIP-L by OpenAI. :)

@zer0int Do you think this would improved text even more if we trained an object with text using the CLIP for training AND Image generation?

Vs just using the OG clip for training and this for image gen?

Assuming you mean "training CLIP and the diffusion model together": Yes, technically, that is possible. However, it's not the best approach, considering CLIP does contrastive learning (the more negative examples there are in a batch => the bigger the batch_size, the better [up to a certain point]). In my experience, it is best to:

1. Fine-tune the CLIP model in full, text and image, standalone
2. Use the CLIP fine-tune with the diffusion model for training, but KEEP CLIP FROZEN and only train the diffusion model for aligning with the fine-tuned CLIP.

Hope that helps!