Initial commit with model files

.gitattributes

@@ -0,0 +1,7 @@
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.tar.gz filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,85 @@
+---
+datasets:
+- yuvalkirstain/pickapic_v2
+language:
+- en
+base_model:
+- stabilityai/stable-diffusion-2-1
+pipeline_tag: text-to-image
+library_name: diffusers
+---
+
+
+---
+# Anonymize Anyone: Toward Race Fairness in Text-to-Face Synthesis using Simple Preference Optimization in Diffusion Model
+
+For detailed information, code, and documentation, please visit our GitHub repository:
+[Anonymize-Anyone](https://github.com/fh2c1/Anonymize-Anyone)
+
+## Anonymize Anyone
+
+![anonymiza-anyone demo images](./assets/Fig1.png)
+
+## Model
+
+![overall_structure](./assets/Fig2.png)
+
+**Anonymize Anyone** presents a novel approach to text-to-face synthesis using a Diffusion Model that considers Race Fairness. Our method uses facial segmentation masks to edit specific facial regions, and employs a Stable Diffusion v2 Inpainting model trained on a curated Asian dataset. We introduce two key losses: **ℒ𝐹𝐹𝐸** (Focused Feature Enhancement Loss) to enhance performance with limited data, and **ℒ𝑫𝑰𝑭𝑭** (Difference Loss) to address catastrophic forgetting. Finally, we apply **Simple Preference Optimization** (SimPO) for refined and enhanced image generation.
+
+## Model Checkpoints
+
+- [Anonymize-Anyone (Inpainting model with **ℒ𝐹𝐹𝐸** and **ℒ𝑫𝑰𝑭𝑭**)](https://huggingface.co/fh2c1/Anonymize-Anyone)
+- [SimPO-LoRA (Diffusion model with **Simple Preference Optimization**)](https://huggingface.co/fh2c1/SimPO-LoRA-1.2)
+
+### Using with Diffusers🧨
+
+You can use this model directly with the `diffusers` library:
+
+
+```python
+import torch
+from PIL import Image
+from diffusers import StableDiffusionInpaintPipeline
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+sd_pipe = StableDiffusionInpaintPipeline.from_pretrained(
+    "fh2c1/Anonymize-Anyone",
+    torch_dtype=torch.float16,
+    safety_checker=None,
+).to(device)
+sd_pipe.load_lora_weights("fh2c1/SimPO-LoRA-1.2", adapter_name="SimPO")
+sd_pipe.set_adapters(["SimPO"], adapter_weights=[0.5])
+
+def generate_image(image_path, mask_path, prompt, negative_prompt, pipe, seed):
+    try:
+        in_image = Image.open(image_path)
+        in_mask = Image.open(mask_path)
+    except IOError as e:
+        print(f"Loading error: {e}")
+        return None
+    generator = torch.Generator(device).manual_seed(seed)
+    result = pipe(image=in_image, mask_image=in_mask, prompt=prompt,
+                  negative_prompt=negative_prompt, generator=generator)
+    return result.images[0]
+
+image = '/content/Anonymize-Anyone/data/2.png'
+mask = "/content/Anonymize-Anyone/data/2_mask.png"
+prompt = "he is an asian man."
+seed = 38189219984105
+negative_prompt = "low resolution, ugly, disfigured, ugly, bad, immature, cartoon, anime, 3d, painting, b&w, deformed eyes, low quailty, noise"
+
+try:
+    generated_image = generate_image(image_path=image, mask_path=mask, prompt=prompt,
+                                     negative_prompt=negative_prompt, pipe=sd_pipe, seed=seed)
+except TypeError as e:
+    print(f"TypeError : {e}")
+
+generated_image
+```
+![result](./assets/Fig3.png)
+
+For more detailed usage instructions, including how to prepare segmentation masks and run inference, please refer to our [GitHub repository](https://github.com/fh2c1/Anonymize-Anyone).
+
+## Training
+
+For information on how to train the model, including the use of **ℒ𝐹𝐹𝐸** (Focused Feature Enhancement Loss) and **ℒ𝑫𝑰𝑭𝑭** (Difference Loss), please see our GitHub repository's [training section](https://github.com/fh2c1/Anonymize-Anyone#running_man-train).

optimizer.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a578a5d4d1b0615f7b81426b14998056200e9827a9245cc5d84977296886fb30
+size 4654586

pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1c06e7534b499a1b6d4647bc4b805379840ea84b5280881fd602058cd9f7d61
+size 6677176

scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1d64a986cf16698550eb29cdcaebce016e0124c48d0224004513d5914a56639
+size 988

scheduler.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9b60b4f92ec11f63bd7ec5d81951de7e3bf250074d6d2864be907c49fb0011b
+size 1000