|
2025-07-01 08:49:26.897 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:26.897 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:26.906 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.199 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.200 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.208 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.263 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.263 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.268 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.268 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.271 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.277 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.277 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.277 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.285 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.318 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.318 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.321 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.321 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.321 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.321 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.321 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.321 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.324 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.324 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.327 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.329 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.330 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.330 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.332 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.396 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.397 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.406 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.457 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.457 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.457 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.457 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.466 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.466 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.516 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.516 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.525 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.533 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.533 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.536 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.536 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.541 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.544 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.562 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.562 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.562 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.562 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.563 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.563 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.564 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.564 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.565 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.565 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.570 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.571 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.572 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.572 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.573 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.611 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.611 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.612 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.612 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.614 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.614 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.620 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.620 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.620 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.621 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.623 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.628 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:27.962 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:27.962 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:27.971 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.089 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.089 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.089 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.089 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.090 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.090 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.092 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.092 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.095 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.095 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.097 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.098 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.099 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.099 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.099 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.101 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.104 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.107 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.153 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.153 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.162 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.186 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.186 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.188 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.189 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.195 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.197 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.202 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.203 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.211 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.259 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.259 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.262 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.262 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.262 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.262 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.264 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.264 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.267 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.270 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.271 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.273 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.283 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.284 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.293 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.333 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.333 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.342 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
2025-07-01 08:49:28.461 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.461 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.462 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.462 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.470 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.470 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.470 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.471 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.472 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.472 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.474 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.474 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.476 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:28.477 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:28.479 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.481 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.482 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:28.485 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
2025-07-01 08:49:29.307 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.307 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.316 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.344 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.344 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.346 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.346 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.353 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.354 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.357 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.357 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.366 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.383 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.383 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
2025-07-01 08:49:29.392 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.396 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.396 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.404 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.469 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.470 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.470 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:29.470 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:29.478 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:29.479 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
2025-07-01 08:49:30.224 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.224 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.225 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.225 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.226 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.226 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.233 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:30.234 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:30.234 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
2025-07-01 08:49:30.336 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.336 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.343 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.343 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.344 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.344 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.345 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:30.347 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.347 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.351 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:30.352 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
2025-07-01 08:49:30.354 | INFO | llava.data.builder:register_datasets:39 - Registering datasets from environment: 'nvidia,draco-oci-iad'. |
|
2025-07-01 08:49:30.354 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/nvidia.yaml'. |
|
2025-07-01 08:49:30.356 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
2025-07-01 08:49:30.362 | INFO | llava.data.builder:register_datasets:44 - Registering datasets from: '/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/data/registry/datasets/draco-oci-iad.yaml'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers |
|
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Did not find AutoResume SDK! |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8171.48it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6443.33it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9133.18it/s] |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/training_args.py:1559: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead |
|
warnings.warn( |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8398.21it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8354.39it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9250.20it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7001.62it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8820.39it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8888.03it/s] |
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8702.74it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8028.47it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 14234.06it/s] |
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9618.91it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 15512.57it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7364.58it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8148.05it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 12403.94it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9556.30it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9138.87it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8005.85it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8008.76it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6628.57it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6799.47it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8423.10it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7399.85it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9357.31it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8933.10it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9489.38it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8241.83it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9035.74it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6578.56it/s] |
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7580.72it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9263.82it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 12800.52it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9826.01it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7292.03it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 13542.49it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6494.65it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8050.49it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8511.00it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7551.47it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8252.64it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8363.12it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6541.43it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7718.23it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9308.85it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6838.01it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7937.31it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6990.51it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6233.13it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8495.41it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7852.40it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9005.25it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8242.60it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 9275.52it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7375.68it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7935.17it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7538.55it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8167.69it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 6128.18it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8195.81it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 5937.34it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 7587.90it/s] |
|
Fetching 21 files: 0%| | 0/21 [00:00<?, ?it/s]
Fetching 21 files: 100%|██████████| 21/21 [00:00<00:00, 8093.39it/s] |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.42s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.42s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.42s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.35s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.35s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.46s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.46s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.45s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.45s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.24s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.26s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.26s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.24s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.24s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.29s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.41s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.42s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.52s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.49s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.40s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.43s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 2
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.42s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.54s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.50s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.51s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.42s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:07, 2.49s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.49s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.39s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.36s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.37s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.41s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.42s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.42s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.27s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.27s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:06, 2.30s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.30s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.31s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.34s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.25s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.25s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:07, 2.42s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.39s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.39s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.39s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.39s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.27s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.30s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.30s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.30s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it] |
|
5%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.72s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.38s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:05<00:05, 2.57s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.24s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.40s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.74s/it] |
|
5%|██▌ | 1/4 [00:02<00:07, 2.52s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.52s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.38s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.38s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.42s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:07, 2.44s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.52s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.39s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.38s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████5%|██▌ | 1/4 [00:02<00:07, 2.54s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.53s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.40s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.38s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:04<00:04, 2.43s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.27s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it] |
|
▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.30s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it] |
|
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.78s/it] |
|
▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.30s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.78s/it] |
|
▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.30s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.30s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
▌ | 3/4 [00:06<00:02, 2.25s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.25s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.25s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.26s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.26s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.26s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.74s/it] |
|
▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.32s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.74s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.74s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.74s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.74s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.78s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.78s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.47s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.80s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:06<00:00, 1.75s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.75s/it] |
|
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:06<00:02, 2.29s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.31s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.78s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.48s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.48s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.48s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.48s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.48s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.82s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.47s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.77s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.82s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
|
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.81s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.83s/it] |
|
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.52s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:07<00:00, 1.85s/it] |
|
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:07<00:02, 2.43s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:08<00:00, 1.73s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:08<00:00, 2.03s/it] |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False` |
|
2025-07-01 08:52:31.220 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.276 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.277 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.342 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.439 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.477 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.485 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.519 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.521 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.523 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.551 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.613 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.634 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.655 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.669 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.670 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.709 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.731 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.733 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.764 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.816 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.845 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.854 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.859 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.863 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.869 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.871 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.881 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.882 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.882 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.893 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.903 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.904 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.923 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.932 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.932 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.934 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.939 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.944 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.950 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.955 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.960 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.969 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.975 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:31.978 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.986 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:31.994 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.007 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.008 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:32.009 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:32.014 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.020 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.027 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.045 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:32.045 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:32.047 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.054 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.073 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.081 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.098 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.129 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:32.133 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
2025-07-01 08:52:32.137 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
2025-07-01 08:52:32.337 | WARNING | llava.data.builder:build_dataset:92 - Training VILA with mixture 'vila_quantumn_qa_train'. |
|
/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py:592: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LLaVATrainer.__init__`. Use `processing_class` instead. |
|
super().__init__(*args, **kwargs) |
|
[rank0]: Traceback (most recent call last): |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/train_mem.py", line 49, in <module> |
|
[rank0]: train() |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/train.py", line 906, in train |
|
[rank0]: trainer.train(resume_from_checkpoint=resume_from_checkpoint) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train |
|
[rank0]: return inner_training_loop( |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/llava/train/llava_trainer.py", line 665, in _inner_training_loop |
|
[rank0]: return super()._inner_training_loop(batch_size, *args, **kwargs) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/trainer.py", line 2381, in _inner_training_loop |
|
[rank0]: self.control = self.callback_handler.on_train_begin(args, self.state, self.control) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 468, in on_train_begin |
|
[rank0]: return self.call_event("on_train_begin", args, state, control) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 518, in call_event |
|
[rank0]: result = getattr(callback, event)( |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 911, in on_train_begin |
|
[rank0]: self.setup(args, state, model, **kwargs) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 838, in setup |
|
[rank0]: self._wandb.init( |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1623, in init |
|
[rank0]: wandb._sentry.reraise(e) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/wandb/analytics/sentry.py", line 156, in reraise |
|
[rank0]: raise exc.with_traceback(sys.exc_info()[2]) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1551, in init |
|
[rank0]: wi.maybe_login(init_settings) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 191, in maybe_login |
|
[rank0]: wandb_login._login( |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/wandb/sdk/wandb_login.py", line 315, in _login |
|
[rank0]: key, key_status = wlogin.prompt_api_key(referrer=referrer) |
|
[rank0]: File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/wandb/sdk/wandb_login.py", line 243, in prompt_api_key |
|
[rank0]: raise UsageError("api_key not configured (no-tty). call " + directive) |
|
[rank0]: wandb.errors.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key]) |
|
W0701 08:52:50.516000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086291 closing signal SIGTERM |
|
W0701 08:52:50.517000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086292 closing signal SIGTERM |
|
W0701 08:52:50.519000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086293 closing signal SIGTERM |
|
W0701 08:52:50.520000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086294 closing signal SIGTERM |
|
W0701 08:52:50.521000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086295 closing signal SIGTERM |
|
W0701 08:52:50.522000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086296 closing signal SIGTERM |
|
W0701 08:52:50.524000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2086297 closing signal SIGTERM |
|
E0701 08:52:54.370000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 2086290) of binary: /lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/bin/python3 |
|
Traceback (most recent call last): |
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/bin/torchrun", line 10, in <module> |
|
sys.exit(main()) |
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper |
|
return f(*args, **kwargs) |
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main |
|
run(args) |
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run |
|
elastic_launch( |
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ |
|
return launch_agent(self._config, self._entrypoint, list(args)) |
|
File "/lustre/fs1/portfolios/nvr/projects/nvr_elm_llm/users/ligengz/workspace/vila-q/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent |
|
raise ChildFailedError( |
|
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: |
|
============================================================ |
|
llava/train/train_mem.py FAILED |
|
------------------------------------------------------------ |
|
Failures: |
|
<NO_OTHER_FAILURES> |
|
------------------------------------------------------------ |
|
Root Cause (first observed failure): |
|
[0]: |
|
time : 2025-07-01_08:52:50 |
|
host : pool0-01504.cm.cluster |
|
rank : 0 (local_rank: 0) |
|
exitcode : 1 (pid: 2086290) |
|
error_file: <N/A> |
|
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
|
============================================================ |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642507 closing signal SIGTERM |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080465 closing signal SIGTERM |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860941 closing signal SIGTERM |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512471 closing signal SIGTERM |
|
W0701 08:52:54.885000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.886000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896464 closing signal SIGTERM |
|
W0701 08:52:54.884000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.885000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139986 closing signal SIGTERM |
|
W0701 08:52:54.885000 23456244200576 torch/distributed/elastic/agent/server/api.py:741] Received Signals.SIGTERM death signal, shutting down workers |
|
W0701 08:52:54.885000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139987 closing signal SIGTERM |
|
W0701 08:52:54.885000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008516 closing signal SIGTERM |
|
W0701 08:52:54.885000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642508 closing signal SIGTERM |
|
W0701 08:52:54.886000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860942 closing signal SIGTERM |
|
W0701 08:52:54.886000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512472 closing signal SIGTERM |
|
W0701 08:52:54.886000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080466 closing signal SIGTERM |
|
W0701 08:52:54.886000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008517 closing signal SIGTERM |
|
W0701 08:52:54.887000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896465 closing signal SIGTERM |
|
W0701 08:52:54.887000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642509 closing signal SIGTERM |
|
W0701 08:52:54.887000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860943 closing signal SIGTERM |
|
W0701 08:52:54.887000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080467 closing signal SIGTERM |
|
W0701 08:52:54.888000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512473 closing signal SIGTERM |
|
W0701 08:52:54.889000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896466 closing signal SIGTERM |
|
W0701 08:52:54.888000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008518 closing signal SIGTERM |
|
W0701 08:52:54.888000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642510 closing signal SIGTERM |
|
W0701 08:52:54.888000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139988 closing signal SIGTERM |
|
W0701 08:52:54.889000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860944 closing signal SIGTERM |
|
W0701 08:52:54.890000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896467 closing signal SIGTERM |
|
W0701 08:52:54.889000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008519 closing signal SIGTERM |
|
W0701 08:52:54.889000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512474 closing signal SIGTERM |
|
W0701 08:52:54.889000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080468 closing signal SIGTERM |
|
W0701 08:52:54.889000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642511 closing signal SIGTERM |
|
W0701 08:52:54.890000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860945 closing signal SIGTERM |
|
W0701 08:52:54.890000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139989 closing signal SIGTERM |
|
W0701 08:52:54.891000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896468 closing signal SIGTERM |
|
W0701 08:52:54.890000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512475 closing signal SIGTERM |
|
W0701 08:52:54.891000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080469 closing signal SIGTERM |
|
W0701 08:52:54.891000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642512 closing signal SIGTERM |
|
W0701 08:52:54.891000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139990 closing signal SIGTERM |
|
W0701 08:52:54.893000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896469 closing signal SIGTERM |
|
W0701 08:52:54.891000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008520 closing signal SIGTERM |
|
W0701 08:52:54.892000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860946 closing signal SIGTERM |
|
W0701 08:52:54.892000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080470 closing signal SIGTERM |
|
W0701 08:52:54.892000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512476 closing signal SIGTERM |
|
W0701 08:52:54.892000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139991 closing signal SIGTERM |
|
W0701 08:52:54.893000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008521 closing signal SIGTERM |
|
W0701 08:52:54.893000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642513 closing signal SIGTERM |
|
W0701 08:52:54.894000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896470 closing signal SIGTERM |
|
W0701 08:52:54.893000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860947 closing signal SIGTERM |
|
W0701 08:52:54.894000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512477 closing signal SIGTERM |
|
W0701 08:52:54.894000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080471 closing signal SIGTERM |
|
W0701 08:52:54.894000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139992 closing signal SIGTERM |
|
W0701 08:52:54.894000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 642514 closing signal SIGTERM |
|
W0701 08:52:54.895000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008522 closing signal SIGTERM |
|
W0701 08:52:54.897000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1896471 closing signal SIGTERM |
|
W0701 08:52:54.896000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1080472 closing signal SIGTERM |
|
W0701 08:52:54.896000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1860948 closing signal SIGTERM |
|
W0701 08:52:54.896000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3008523 closing signal SIGTERM |
|
W0701 08:52:54.896000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1512478 closing signal SIGTERM |
|
W0701 08:52:54.897000 23456244200576 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 139993 closing signal SIGTERM |
|
|