|
|
2025-07-01 09:15:45.264 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:45.265 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:45.273 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:45.395 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:45.395 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:45.404 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:46.045 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:46.045 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:46.054 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:46.068 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:46.068 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:46.077 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:46.156 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:46.156 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:46.165 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:46.191 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:46.191 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:46.196 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:46.196 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:46.197 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:46.197 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:46.200 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:46.204 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:46.206 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
Did not find AutoResume SDK! |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 11409.38it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8468.45it/s] |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.649 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.651 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.651 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.651 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.651 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.653 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.654 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.659 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.663 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.663 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.663 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.663 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.663 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.663 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.664 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.664 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.668 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.668 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.669 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
2025-07-01 09:15:50.795 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.795 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.795 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.795 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.795 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.795 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.796 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.796 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.796 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.796 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.796 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.796 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.797 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.797 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.797 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.797 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.805 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.805 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.805 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.805 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.805 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.805 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.814 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.814 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.867 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.868 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:50.877 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.877 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.877 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.878 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.878 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.878 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.878 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:50.878 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.109 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.109 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.109 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.109 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.109 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.109 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.110 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.110 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.110 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.110 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.110 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.110 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.111 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.111 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:51.111 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.111 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:51.119 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.119 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.119 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.119 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.119 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.119 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.128 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:51.128 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8222.59it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8411.04it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9788.88it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 10111.40it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8872.81it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8404.62it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
2025-07-01 09:15:52.817 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.817 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.818 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
2025-07-01 09:15:52.828 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Did not find AutoResume SDK! |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 5913.42it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6102.29it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6171.98it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6194.12it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 5129.30it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 5824.27it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 11223.29it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7932.31it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7381.24it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 11346.18it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7184.37it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7508.34it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 10911.84it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7567.04it/s] |
|
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8954.90it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7803.01it/s] |
|
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8492.95it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7792.66it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7313.82it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8479.05it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8502.79it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8082.25it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8943.08it/s] |
|
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7816.86it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9318.70it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 10563.73it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7557.95it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7073.59it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7256.58it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6846.51it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9073.90it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7825.89it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9962.72it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6033.32it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 10411.39it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8170.72it/s] |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
|
warnings.warn( |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 10455.89it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8244.14it/s] |
|
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8419.88it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7635.26it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8344.90it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6743.25it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7058.85it/s] |
|
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6877.52it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8550.66it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 10852.68it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9270.64it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 11510.77it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 5884.97it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 11163.55it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 11005.92it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7496.84it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8553.99it/s] |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8777.32it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9354.33it/s] |
|
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8173.76it/s] |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.79s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.79s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.81s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.93s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.93s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.98s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.95s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.99s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.73s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.66s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.75s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.75s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.82s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.75s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.65s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.76s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.80s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.81s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.75s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.79s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.49s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.57s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.54s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.58s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.58s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.37s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:08, 2.86s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.86s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.59s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.62s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:08, 2.84s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.86s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.62s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:08, 2.82s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.82s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.52s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.59s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.58s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.60s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.61s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.36s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.47s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.84s/it] |
|
|
5%|██▌ | 1/4 [00:02<00:08, 2.99s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.14s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.66s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.68s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.66s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.68s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.68s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.70s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.78s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.43s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.58s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.63s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.63s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.63s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.63s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.87s/it] |
|
|
5%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.55s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.48s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.50s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.30s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.33s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
|
5%|██▌ | 1/4 [00:02<00:08, 2.90s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.89s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.62s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.64s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.41s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.89s/it] |
|
|
▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.88s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.89s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.88s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.88s/it] |
|
|
▌ | 3/4 [00:07<00:02, 2.44s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.44s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.44s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.44s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.54s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.93s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.90s/it] |
|
|
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.35s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.35s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.35s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.35s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.96s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.96s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.96s/it] |
|
|
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.41s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.40s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.54s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.91s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.93s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.91s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.93s/it] |
|
|
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.93s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.92s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it] |
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.94s/it] |
|
|
▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.42s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.93s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.95s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.86s/it] |
|
|
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.54s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:08<00:00, 1.79s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:08<00:00, 2.14s/it] |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
|
2025-07-01 09:19:03.830 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:03.890 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:03.923 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:03.948 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:03.959 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:03.965 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.032 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.144 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.189 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.190 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.194 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.217 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.251 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.282 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.285 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.294 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.297 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.313 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.324 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.332 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.339 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.346 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.356 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.365 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.367 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.379 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.382 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.384 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.387 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.399 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.401 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.402 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.407 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.409 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.418 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.428 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.429 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.433 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.436 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.438 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.447 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.449 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.459 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.459 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.464 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.470 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.471 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.472 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.474 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.476 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.484 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.490 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.495 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.509 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.520 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.533 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.533 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.540 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.541 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.541 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.571 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.586 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
2025-07-01 09:19:04.596 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
2025-07-01 09:19:04.621 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
|
super().__init__(*args, **kwargs) |
|
|
wandb: Currently logged in as: ligeng-zhu to https://api.wandb.ai. Use `wandb login --relogin` to force relogin |
|
|
wandb: creating run |
|
|
wandb: Tracking run with wandb version 0.20.1 |
|
|
wandb: Run data is saved locally in runs/train/NVILA-Lite-8B-quantumn-qa-train/wandb/run-20250701_091917-NVILA-Lite-8B-quantumn-qa-train |
|
|
wandb: Run `wandb offline` to turn off syncing. |
|
|
wandb: Resuming run NVILA-Lite-8B-quantumn-qa-train |
|
|
wandb: ⭐️ View project at https://wandb.ai/ligeng-zhu/vila |
|
|
wandb: 🚀 View run at https://wandb.ai/ligeng-zhu/vila/runs/NVILA-Lite-8B-quantumn-qa-train |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
0%| | 0/7 [00:00<?, ?it/s]/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/model/llava_arch.py:534: UserWarning: Truncating sequences to `model_max_length` (4096). |
|
|
warnings.warn(f"Truncating sequences to `model_max_length` ({self.tokenizer.model_max_length}).") |
|
|
[rank63]: Traceback (most recent call last): |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/train_mem.py", line 49, in <module> |
|
|
[rank63]: train() |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/train.py", line 906, in train |
|
|
[rank63]: trainer.train(resume_from_checkpoint=resume_from_checkpoint) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train |
|
|
[rank63]: return inner_training_loop( |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py", line 665, in _inner_training_loop |
|
|
[rank63]: return super()._inner_training_loop(batch_size, *args, **kwargs) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop |
|
|
[rank63]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/transformer_normalize_monkey_patch.py", line 247, in training_step |
|
|
[rank63]: self.accelerator.backward(loss, **kwargs) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 2188, in backward |
|
|
[rank63]: self.deepspeed_engine_wrapped.backward(loss, **kwargs) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 166, in backward |
|
|
[rank63]: self.engine.backward(loss, **kwargs) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn |
|
|
[rank63]: ret_val = func(*args, **kwargs) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1861, in backward |
|
|
[rank63]: self.optimizer.backward(loss, retain_graph=retain_graph) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn |
|
|
[rank63]: ret_val = func(*args, **kwargs) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 1993, in backward |
|
|
[rank63]: self.loss_scaler.backward(loss.float(), retain_graph=retain_graph) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward |
|
|
[rank63]: scaled_loss.backward(retain_graph=retain_graph) |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward |
|
|
[rank63]: torch.autograd.backward( |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward |
|
|
[rank63]: _engine_run_backward( |
|
|
[rank63]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward |
|
|
[rank63]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass |
|
|
[rank63]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 15.07 GiB. GPU has a total capacity of 79.10 GiB of which 14.49 GiB is free. Including non-PyTorch memory, this process has 64.58 GiB memory in use. Of the allocated memory 56.11 GiB is allocated by PyTorch, and 2.37 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) |
|
|
W0701 09:19:49.372000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163027 closing signal SIGTERM |
|
|
W0701 09:19:49.374000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163028 closing signal SIGTERM |
|
|
W0701 09:19:49.375000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163029 closing signal SIGTERM |
|
|
W0701 09:19:49.376000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163030 closing signal SIGTERM |
|
|
W0701 09:19:49.378000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163031 closing signal SIGTERM |
|
|
W0701 09:19:49.379000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163032 closing signal SIGTERM |
|
|
W0701 09:19:49.381000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 163033 closing signal SIGTERM |
|
|
E0701 09:20:02.642000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 7 (pid: 163034) of binary: /lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/bin/python3 |
|
|
Traceback (most recent call last): |
|
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/bin/torchrun", line 10, in <module> |
|
|
sys.exit(main()) |
|
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper |
|
|
return f(*args, **kwargs) |
|
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main |
|
|
run(args) |
|
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run |
|
|
elastic_launch( |
|
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ |
|
|
return launch_agent(self._config, self._entrypoint, list(args)) |
|
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent |
|
|
raise ChildFailedError( |
|
|
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: |
|
|
============================================================ |
|
|
llava/train/train_mem.py FAILED |
|
|
------------------------------------------------------------ |
|
|
Failures: |
|
|
<NO_OTHER_FAILURES> |
|
|
------------------------------------------------------------ |
|
|
Root Cause (first observed failure): |
|
|
[0]: |
|
|
time : 2025-07-01_09:19:49 |
|
|
host : pool0-01939.cm.cluster |
|
|
rank : 63 (local_rank: 7) |
|
|
exitcode : 1 (pid: 163034) |
|
|
error_file: <N/A> |
|
|
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
|
|
============================================================ |
|
|
Jul 01 09:20:03.175064 2126192 slurmstepd 0x1555337b7640: error: *** STEP 1038294.0 ON pool0-01504 CANCELLED AT 2025-07-01T09:20:03 *** |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429699 closing signal SIGTERM |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849712 closing signal SIGTERM |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702325 closing signal SIGTERM |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886519 closing signal SIGTERM |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702326 closing signal SIGTERM |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126487 closing signal SIGTERM |
|
|
W0701 09:20:03.178000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846575 closing signal SIGTERM |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
|
W0701 09:20:03.177000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650565 closing signal SIGTERM |
|
|
W0701 09:20:03.178000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429700 closing signal SIGTERM |
|
|
W0701 09:20:03.178000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886520 closing signal SIGTERM |
|
|
W0701 09:20:03.179000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849713 closing signal SIGTERM |
|
|
W0701 09:20:03.178000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126488 closing signal SIGTERM |
|
|
W0701 09:20:03.179000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846576 closing signal SIGTERM |
|
|
W0701 09:20:03.179000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702327 closing signal SIGTERM |
|
|
W0701 09:20:03.179000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650566 closing signal SIGTERM |
|
|
W0701 09:20:03.180000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429701 closing signal SIGTERM |
|
|
W0701 09:20:03.180000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849714 closing signal SIGTERM |
|
|
W0701 09:20:03.180000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846577 closing signal SIGTERM |
|
|
W0701 09:20:03.180000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886521 closing signal SIGTERM |
|
|
W0701 09:20:03.180000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126490 closing signal SIGTERM |
|
|
W0701 09:20:03.180000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650567 closing signal SIGTERM |
|
|
W0701 09:20:03.182000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849715 closing signal SIGTERM |
|
|
W0701 09:20:03.181000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702328 closing signal SIGTERM |
|
|
W0701 09:20:03.181000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846578 closing signal SIGTERM |
|
|
W0701 09:20:03.181000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429702 closing signal SIGTERM |
|
|
W0701 09:20:03.181000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886522 closing signal SIGTERM |
|
|
W0701 09:20:03.181000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126492 closing signal SIGTERM |
|
|
W0701 09:20:03.182000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650568 closing signal SIGTERM |
|
|
W0701 09:20:03.183000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429703 closing signal SIGTERM |
|
|
W0701 09:20:03.183000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849716 closing signal SIGTERM |
|
|
W0701 09:20:03.183000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126499 closing signal SIGTERM |
|
|
W0701 09:20:03.183000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846579 closing signal SIGTERM |
|
|
W0701 09:20:03.183000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886523 closing signal SIGTERM |
|
|
W0701 09:20:03.183000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650569 closing signal SIGTERM |
|
|
W0701 09:20:03.184000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846580 closing signal SIGTERM |
|
|
W0701 09:20:03.184000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702329 closing signal SIGTERM |
|
|
W0701 09:20:03.185000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849717 closing signal SIGTERM |
|
|
W0701 09:20:03.184000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126501 closing signal SIGTERM |
|
|
W0701 09:20:03.185000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650570 closing signal SIGTERM |
|
|
W0701 09:20:03.185000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886524 closing signal SIGTERM |
|
|
W0701 09:20:03.185000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429704 closing signal SIGTERM |
|
|
W0701 09:20:03.185000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702330 closing signal SIGTERM |
|
|
W0701 09:20:03.185000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126502 closing signal SIGTERM |
|
|
W0701 09:20:03.186000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849718 closing signal SIGTERM |
|
|
W0701 09:20:03.186000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846581 closing signal SIGTERM |
|
|
W0701 09:20:03.186000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650571 closing signal SIGTERM |
|
|
W0701 09:20:03.187000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429705 closing signal SIGTERM |
|
|
W0701 09:20:03.187000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886525 closing signal SIGTERM |
|
|
W0701 09:20:03.188000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1846582 closing signal SIGTERM |
|
|
W0701 09:20:03.187000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2126503 closing signal SIGTERM |
|
|
W0701 09:20:03.188000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2849719 closing signal SIGTERM |
|
|
W0701 09:20:03.188000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702331 closing signal SIGTERM |
|
|
W0701 09:20:03.189000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1429706 closing signal SIGTERM |
|
|
W0701 09:20:03.188000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 650572 closing signal SIGTERM |
|
|
W0701 09:20:03.189000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 886526 closing signal SIGTERM |
|
|
W0701 09:20:03.189000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2702332 closing signal SIGTERM |
|
|
|
