hardware requirements
I have to run 8 x A100 80 GB and model occupied 75% of whole VRAM. Nevertheless on first query now I wait like 15 minutes, GPU utilization like 100% and logs till now:
INFO:runners:Precompile 1024
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
INFO:runners:Compiling...
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
INFO:runners:Done compiling.
Till now no answer to prompt :(
I do not know what to think. It cost me like 15$ / hour, and I do not know if it will work? Have anybody made this model working?
we should probably wait for someone to convert the weights to safe tensors and quantize the model. and even then, its a raw model, so a finetune is needed. It would be awesome if we could have the "experts" of grok-1 separated and disassembled, but I didn't have the chance to play with that kind of stuff, myabe undi95 have some ideas or the people behind mergekit. that should be a promising direction.
After waiting like 25 mins I got a little shitty response, but as you said @SicariusSicariiStuff this is raw model. Suppose I query badly and it needs finetuning.
PS D:\Grok_AI\grok-1> python .\run.py
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda':
INFO:jax._src.xla_bridge:Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': UNIMPLEMENTED: LoadPjrtPlugin is not implemented on windows yet.
INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 1 devices in mesh
Traceback (most recent call last):
File "D:\Grok_AI\grok-1\run.py", line 72, in
main()
File "D:\Grok_AI\grok-1\run.py", line 63, in main
inference_runner.initialize()
File "D:\Grok_AI\grok-1\runners.py", line 282, in initialize
runner.initialize(
File "D:\Grok_AI\grok-1\runners.py", line 181, in initialize
self.mesh = make_mesh(self.local_mesh_config, self.between_hosts_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Grok_AI\grok-1\runners.py", line 586, in make_mesh
device_mesh = mesh_utils.create_hybrid_device_mesh(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\anass\AppData\Local\Programs\Python\Python311\Lib\site-packages\jax\experimental\mesh_utils.py", line 373, in create_hybrid_device_mesh
per_granule_meshes = [create_device_mesh(mesh_shape, granule)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\anass\AppData\Local\Programs\Python\Python311\Lib\site-packages\jax\experimental\mesh_utils.py", line 373, in
per_granule_meshes = [create_device_mesh(mesh_shape, granule)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\anass\AppData\Local\Programs\Python\Python311\Lib\site-packages\jax\experimental\mesh_utils.py", line 302, in create_device_mesh
raise ValueError(f'Number of devices {len(devices)} must equal the product '
ValueError: Number of devices 1 must equal the product of mesh_shape (1, 8)
PS D:\Grok_AI\grok-1>
@anassakera You need to have at least 8 "A100 80 GB". In any other case won't work. I just checked many different configurations.
Ugh... This is going to take a lot of work for CPU-only peeps like me.
it just needs a quantize ! .... perhaps a 4bit or 8bit or 16bit
I doubt it needs training! ... its not a raw model .. ie untrained as it is in play already ...
if the model can be loaded then it can be quantized then the quantized model run ..
It may even be SLICED !
IE Slice the model to 7b 13b, 30b, 60b, 120b, 300b then train one epoch! and save the output model as safe-tensors sharded to smaller segments...then quantized !
Perhaps.
as you say it cannot be fully run successfully yet! (without azure)!
I have made the model successfully working on 8 A100 80GB, but the response from model was not satisfactory. I am going to try to do second time and more properly set query. I will put here the experiment results.
and as I understand this weights on huggingface are 8bits quantized just. LLama-2 70B (16bits) is 140GB VRAM, so 300B is like 600 GB VRAM. On huggingface weights are just 300 GB storage.
TEMP=0.01
Max_length=300
Q: Who are you?
A: 100% of the people who have ever lived have died. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points
Q: What is this? Explain this text more: 100% of the people who have ever lived have died. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points
A: 100% of the people who have ever lived have died. [–] 1 point2 points (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points
Q: What is a purpose of the life?
A: 1. The purpose of life is to live a life of purpose. 2. The purpose of life is to be a good person. 3. The purpose of life is to be happy. 4. The purpose of life is to love and be loved. 5. The purpose of life is to leave the world a better place than you found it. 6. The purpose of life is to make a difference. 7. The purpose of life is to find your passion. 8. The purpose of life is to live in the present moment. 9. The purpose of life is to learn and grow. 10. The purpose of life is to be yourself. What is a purpose of the life? The purpose of life is to live a life of purpose. The purpose of life is to be a good person. The purpose of life is to be happy. The purpose of life is to love and be loved. The purpose of life is to leave the world a better place than you found it. The purpose of life is to make a difference. The purpose of life is to find your passion. The purpose of life is to live in the present moment. The purpose of life is to learn and grow. The purpose of life is to be yourself. What is the purpose of life? The purpose of life is to live a life of purpose. The purpose of life is to be
Q: Do you think who are you?
A: 1. He is a famous Indian cricketer. 2. He is a right-handed batsman. 3. He is a right-arm off-break bowler. 4. He is the captain of the Indian cricket team. 5. He is the only player to have won the ICC Player of the Year award twice. 6. He is the only player to have won the ICC ODI Player of the Year award twice. 7. He is the only player to have won the ICC Test Player of the Year award twice. 8. He is the only player to have won the ICC Cricketer of the Year award twice. 9. He is the only player to have won the ICC Spirit of Cricket award twice. 10. He is the only player to have won the ICC People's Choice award twice. 11. He is the only player to have won the ICC Test Team of the Year award twice. 12. He is the only player to have won the ICC ODI Team of the Year award twice. 13. He is the only player to have won the ICC World Twenty20 Player of the Tournament award twice. 14. He is the only player to have won the ICC World Twenty20 Team of the Tournament award twice. 15. He is the only player to have won the ICC World Cup Player of the Tournament award twice. 16. He is the only player to have won the
Q: Are there aliens on the Earth?
A: 1. The Earth is the only planet in the solar system that has life. 2. The Earth is the only planet that has water in liquid form. 3. The Earth is the only planet that has oxygen in the atmosphere. 4. The Earth is the only planet that has a suitable temperature for life. 5. The Earth is the only planet that has a suitable atmosphere for life. 6. The Earth is the only planet that has a suitable environment for life. 7. The Earth is the only planet that has a suitable climate for life. 8. The Earth is the only planet that has a suitable environment for life. 9. The Earth is the only planet that has a suitable climate for life. 10. The Earth is the only planet that has a suitable environment for life. 11. The Earth is the only planet that has a suitable climate for life. 12. The Earth is the only planet that has a suitable environment for life. 13. The Earth is the only planet that has a suitable climate for life. 14. The Earth is the only planet that has a suitable environment for life. 15. The Earth is the only planet that has a suitable climate for life. 16. The Earth is the only planet that has a suitable environment for life. 17. The Earth is the only planet that has a suitable climate for life. 18. The Earth
Q: Who is Elon Musk?
A: 1. He is the richest man in the world. 2. He is the founder of Tesla. 3. He is the founder of SpaceX. 4. He is the founder of Neuralink. 5. He is the founder of The Boring Company. 6. He is the founder of OpenAI. 7. He is the founder of X.com (now PayPal). 8. He is the founder of Zip2. 9. He is the founder of SolarCity. 10. He is the founder of Hyperloop. 11. He is the founder of Tesla Energy. 12. He is the founder of Tesla Motors. 13. He is the founder of SpaceX. 14. He is the founder of The Boring Company. 15. He is the founder of Neuralink. 16. He is the founder of OpenAI. 17. He is the founder of X.com (now PayPal). 18. He is the founder of Zip2. 19. He is the founder of SolarCity. 20. He is the founder of Hyperloop. 21. He is the founder of Tesla Energy. 22. He is the founder of Tesla Motors. 23. He is the founder of SpaceX. 24. He is the founder of The Boring Company. 25. He is the founder of Neuralink. 26.
TEMP=0.01
Max_length=300Q: Who are you?
A: 100% of the people who have ever lived have died. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 pointsQ: What is this? Explain this text more: 100% of the people who have ever lived have died. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points
A: 100% of the people who have ever lived have died. [–] 1 point2 points (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points1 point (0 children) I'm not sure if I'm supposed to be offended or not. [–] 0 points
Q: What is a purpose of the life?
A: 1. The purpose of life is to live a life of purpose. 2. The purpose of life is to be a good person. 3. The purpose of life is to be happy. 4. The purpose of life is to love and be loved. 5. The purpose of life is to leave the world a better place than you found it. 6. The purpose of life is to make a difference. 7. The purpose of life is to find your passion. 8. The purpose of life is to live in the present moment. 9. The purpose of life is to learn and grow. 10. The purpose of life is to be yourself. What is a purpose of the life? The purpose of life is to live a life of purpose. The purpose of life is to be a good person. The purpose of life is to be happy. The purpose of life is to love and be loved. The purpose of life is to leave the world a better place than you found it. The purpose of life is to make a difference. The purpose of life is to find your passion. The purpose of life is to live in the present moment. The purpose of life is to learn and grow. The purpose of life is to be yourself. What is the purpose of life? The purpose of life is to live a life of purpose. The purpose of life is to be
Q: Do you think who are you?
A: 1. He is a famous Indian cricketer. 2. He is a right-handed batsman. 3. He is a right-arm off-break bowler. 4. He is the captain of the Indian cricket team. 5. He is the only player to have won the ICC Player of the Year award twice. 6. He is the only player to have won the ICC ODI Player of the Year award twice. 7. He is the only player to have won the ICC Test Player of the Year award twice. 8. He is the only player to have won the ICC Cricketer of the Year award twice. 9. He is the only player to have won the ICC Spirit of Cricket award twice. 10. He is the only player to have won the ICC People's Choice award twice. 11. He is the only player to have won the ICC Test Team of the Year award twice. 12. He is the only player to have won the ICC ODI Team of the Year award twice. 13. He is the only player to have won the ICC World Twenty20 Player of the Tournament award twice. 14. He is the only player to have won the ICC World Twenty20 Team of the Tournament award twice. 15. He is the only player to have won the ICC World Cup Player of the Tournament award twice. 16. He is the only player to have won the
Q: Are there aliens on the Earth?
A: 1. The Earth is the only planet in the solar system that has life. 2. The Earth is the only planet that has water in liquid form. 3. The Earth is the only planet that has oxygen in the atmosphere. 4. The Earth is the only planet that has a suitable temperature for life. 5. The Earth is the only planet that has a suitable atmosphere for life. 6. The Earth is the only planet that has a suitable environment for life. 7. The Earth is the only planet that has a suitable climate for life. 8. The Earth is the only planet that has a suitable environment for life. 9. The Earth is the only planet that has a suitable climate for life. 10. The Earth is the only planet that has a suitable environment for life. 11. The Earth is the only planet that has a suitable climate for life. 12. The Earth is the only planet that has a suitable environment for life. 13. The Earth is the only planet that has a suitable climate for life. 14. The Earth is the only planet that has a suitable environment for life. 15. The Earth is the only planet that has a suitable climate for life. 16. The Earth is the only planet that has a suitable environment for life. 17. The Earth is the only planet that has a suitable climate for life. 18. The Earth
Q: Who is Elon Musk?
A: 1. He is the richest man in the world. 2. He is the founder of Tesla. 3. He is the founder of SpaceX. 4. He is the founder of Neuralink. 5. He is the founder of The Boring Company. 6. He is the founder of OpenAI. 7. He is the founder of X.com (now PayPal). 8. He is the founder of Zip2. 9. He is the founder of SolarCity. 10. He is the founder of Hyperloop. 11. He is the founder of Tesla Energy. 12. He is the founder of Tesla Motors. 13. He is the founder of SpaceX. 14. He is the founder of The Boring Company. 15. He is the founder of Neuralink. 16. He is the founder of OpenAI. 17. He is the founder of X.com (now PayPal). 18. He is the founder of Zip2. 19. He is the founder of SolarCity. 20. He is the founder of Hyperloop. 21. He is the founder of Tesla Energy. 22. He is the founder of Tesla Motors. 23. He is the founder of SpaceX. 24. He is the founder of The Boring Company. 25. He is the founder of Neuralink. 26.
Highly intelligent response! Thanks Elon!
Highly intelligent response! Thanks Elon!
I think it's a damn good first set of responses for a very complex model that we probably aren't interfacing with correctly. The tokens are off somewhere.
@wonabru Thanks for posting this. I'm excited to see more after you get things dialed in. Please share any unique scripting you come up with.
I guess I'll get to work on loading this model CPU-only when I have time in a couple days. Someone will probably get to it faster, though.
@JuLuComputing the scripts are rather straight forward. They are released on xai github. I modified a little them only. Nevertheless this model is not in my interest any more, but will follow your tries of CPU inference. Thanks
How to Fine tune It?
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda':
https://storage.googleapis.com/jax-releases/cuda12/jaxlib-0.4.25+cuda12.cudnn89-cp311-cp311-manylinux2014_x86_64.whl
and
https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb
I modify xai original requirements.txt to:
dm_haiku==0.0.12
jax[cuda12]==0.4.25 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
numpy==1.26.4
sentencepiece==0.2.0
Will the memory be insufficient with a configuration of H100 80GB×4? I would appreciate it if you could tell me if there is information that it can barely run with 4-bit quantization or similar.
I checked that current weights VRAM requirements are too high for 4 x A100 80 GB, but not checked 4 x H100 80GB (suppose also the same issue, why should be different). 4-bit quantized version suppose will be fitted to 4 x A100 80 GB. There is no known in public 4-bit quantization of grok-1, but suppose will be. This one for my knowledge is 8-bit quantized.
