Update README.md

README.md

@@ -1,3 +1,74 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+---
+
+
+# MP_Blendshapes
+
+## Model Description
+MP_Blendshapes has been ported to pytorch from Google's mediapipe library Liam Schoneveld's github [repository](https://github.com/nlml/deconstruct-mediapipe). The inputs are 146/473 of mediapipe's FaceMeshV2 high density landmark model. The 52 blendshapes are similar to ARKit and loosely correspond to Facial Action Unit Coding Systen.
+
+See the mediapipe [model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf) for more details.
+
+## Model Details
+- **Model Type**: MLP-Mixer (Keras)
+- **Framework**: pytorch
+
+## Model Sources
+- **Repository**: [GitHub Repository](https://github.com/cosanlab/py-feat)
+- **Paper**: [Mediapipe blendshape model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf)
+
+## Citation
+If you use the mp_blendshapes model in your research or application, please cite the following paper:
+
+Grishchenko, I., Ablavatski, A., Kartynnik, Y., Raveendran, K., & Grundmann, M. (2020). Attention mesh: High-fidelity face mesh prediction in real-time. arXiv preprint arXiv:2006.10962.
+
+```
+@article{grishchenko2020attention,
+  title={Attention mesh: High-fidelity face mesh prediction in real-time},
+  author={Grishchenko, Ivan and Ablavatski, Artsiom and Kartynnik, Yury and Raveendran, Karthik and Grundmann, Matthias},
+  journal={arXiv preprint arXiv:2006.10962},
+  year={2020}
+}
+```
+
+## Example Useage
+
+```python
+import torch
+import pandas as pd
+from huggingface_hub import hf_hub_download
+from feat.au_detectors.MP_Blendshapes.MP_Blendshapes_test import MediaPipeBlendshapesMLPMixer     
+from feat.utils import MP_BLENDSHAPE_MODEL_LANDMARKS_SUBSET, MP_BLENDSHAPE_NAMES
+
+device = 'cpu'
+
+# Load model and weights
+blendshape_detector = MediaPipeBlendshapesMLPMixer()
+model_path = hf_hub_download(repo_id="py-feat/mp_blendshapes", filename="face_blendshapes.pth")
+blendshape_model_file = hf_hub_download(repo_id='py-feat/resmasknet', filename="ResMaskNet_Z_resmasking_dropout1_rot30.pth")
+blendshape_checkpoint = torch.load(blendshape_model_file, map_location=device)["net"]
+blendshape_detector.load_state_dict(blendshape_checkpoint)
+blendshape_detector.eval()
+blendshape_detector.to(device)
+
+
+# Test model
+face_image = "path/to/your/test_image.jpg"  # Replace with your extracted face image that is [224, 224]
+
+# Extract Landmarks
+landmark_detector = torch.load('/Users/lukechang/Dropbox/py-feat/mediapipe/model/face_landmarks_detector_Nx3x256x256_onnx.pth', weights_only=False)
+landmark_detector.eval()
+landmark_detector.to(device)
+landmark_results = landmark_detector(torch.tensor(face_image).to(device))
+
+# Blendshape Classification
+landmarks = landmark_results[0].reshape(1,478,3)[:,:,:2]
+img_size = torch.tensor((face_image_width, face_image_height)).unsqueeze(0).unsqueeze(0)
+landmarks = landmarks * img_size
+blendshapes = blendshape_detector(landmarks)
+blendshape_results = pd.Series(blendshape_results.squeeze().detach().numpy(), index=BLENDSHAPE_NAMES)
+        
+```