Spaces:
Runtime error
Runtime error
File size: 1,195 Bytes
0c8d55e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
The original code is from [GenAI-Bench](https://github.com/linzhiqiu/t2v_metrics).
## Requirements and Installation
```
pip install git+https://github.com/openai/CLIP.git
pip install open-clip-torch
```
## Eval
### Generate samples
We also support `genai1600`, just replace`genai527.yaml` with `genai1600.yaml` and change `$OUTPUT_DIR`.
```bash
# switch to univa env
MODEL_PATH='path/to/model'
OUTPUT_DIR='path/to/eval_output/genai527'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun \
--nproc_per_node 8 \
-m step1_gen_samples \
genai527.yaml \
--pretrained_lvlm_name_or_path ${MODEL_PATH} \
--output_dir ${OUTPUT_DIR}
```
### Evaluation & Summary
Download [zhiqiulin/clip-flant5-xxl](https://huggingface.co/zhiqiulin/clip-flant5-xxl) to `$T5_PATH`.
Download [openai/clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) to `$VISION_TOWER`.
```bash
# switch to univa env
META_DIR="eval_prompts/genai527"
IMAGE_DIR=${OUTPUT_DIR}
CUDA_VISIBLE_DEVICES=4 VISION_TOWER=${VISION_TOWER} python -m step2_run_model \
--model_path ${T5_PATH} \
--image_dir ${IMAGE_DIR} \
--meta_dir ${META_DIR} > ${IMAGE_DIR}.txt
cat ${IMAGE_DIR}.txt
```
|