FantasiaFoundry

/

GGUF-Quantization-Script

like 60

Ah, so it's the huggingface_hub cache causing it. It's caching all downloaded models to the C:\Users\{{User}}\.cache\huggingface\hub folder in the C drive. This can be useful or a hindrance, for example if you try to convert the same model again later and it's in the cache it won't have to be downloaded again, it will just be symlinked and used for quantization. But it can use disk space heavily if you're just doing model after model without checking it. On my end I have a manual PowerShell command using rmdir "C:\Users\{{User}}\.cache\huggingface\hub" aliased in my Terminal to "rmhfcache" which I run after I'm done with conversions for the day to clear that up...

From their documentation:
https://huggingface.co/docs/transformers/installation?highlight=transformers_cache#cache-setup

I suppose you can add something like $env:HF_HOME = 'YOUR_NEW_HF_PATH' to your Terminal $Profile file and it will be set upon its launch.

Added relevant notice to ReadMe.

@Lewdiculous

That's another one, I noticed this one before writing the thread (and I clean it up), but there is something else happening with cublast, and IDK if it's this script doing it, or something else, this is 105GB of cublast DLLs:

I think it could be... kobold cpp doing this on every model launch?! A bit crazy but it may not be this script as the script only downloads changes to llama cpp. It could also be mergekit somehow doing it... I'm not sure, but it's crazy littering xd

And these are some actual hub cache models, another category

Cuda DLLs are checked if they already exist in the bin folder, and their download is skipped if they are already there, basically they should only be downloaded in the first time using the script. Interesting observation with the temp folder.

Okay, so are these the files you're seeing?

This really seems like a KCPP thing, not related to model quants. It might be a good idea to raise an issue about it - my screenshot looks good enough, and link it here so we can boost it.

Running a model now gave me another one already:

I think you are right, I also observed kobold-related naming in some of these folders, and YES, they are named exactly like you showed. Thanks a lot for the support on investigating this, I will raise it in their discord <3

Opened an issue: https://github.com/LostRuins/koboldcpp/issues/768

Opened an issue: https://github.com/LostRuins/koboldcpp/issues/768

It's fixed! I got 30GBs of storage back :3
And maybe now my bandwidth usage will fall? i hope at least

Wait, didn't you say you use the terminal to avoid bloat? :D I will be needing those juicy 100mb vram nom nom

I have my priorities well defined, as you can see a man can't compromise on everything.

The dedication of these devs is insane, like it must be so much harder to diagnose stuff now that there is like 20-30 different total quant types and each of their versions 😭

I believe some consolidation of Quant options is planned in Llama.Cpp, to bring the number of quants down and help users choose the best performing options for them easier... Eventually. Surely. Soon™.

If llama3's release date lands according to plan llamacpp devs will probably be busy building compatibility😭

"Within the next month, actually less, hopefully in a very short period of time, we hope to start rolling out our new suite of next-generation foundation models, Llama 3,”

A week ago @_@

Edit - source

4 days left, or riot

I sure hope they aren't using emu image generation for meta-ai on Instagram its uh bad 😭
Prompted with "1970's living room"

Meta-AI

JuggernautXL (SDXL)

Marking as solved. Have fun!