Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
julien-cΒ 
posted an update 14 days ago
Post
7546
After some heated discussion πŸ”₯, we clarify our intent re. storage limits on the Hub

TL;DR:
- public storage is free, and (unless blatant abuse) unlimited. We do ask that you consider upgrading to PRO and/or Enterprise Hub if possible
- private storage is paid above a significant free tier (1TB if you have a paid account, 100GB otherwise)

docs: https://huggingface.co/docs/hub/storage-limits

We optimize our infrastructure continuously to scale our storage for the coming years of growth in Machine learning, to the benefit of the community πŸ”₯

cc: @reach-vb @pierric @victor and the HF team

Finally, a happy ending

This makes perfect sense, average users definitely don't need to be uploading that much stuff privately, great for testing but if it's not worth releasing publicly it's not worth storing on servers for free :)

Great update !

btw we’ll continue to support the community via our grants programme, you can read more about it here: https://huggingface.co/docs/hub/storage-limits

image.png

Once the new rules go live, what will happen to the accounts that already have private repos exceeding the 100GB mark?

Β·

For existing users/orgs, the limits come on top of the storage already used πŸ€—

I really like this πŸ€— .

image.png

Amazing!!!

I was about to delete some of my models and datasets to save space. What a relief.

Anyway, as a way for users to deal with this, it seems that if we are unsure, we should just leave it as Public.
We should only use Private for things that are truly private or for work in progress.
Thank you for your generous treatment.😊

Perhaps HF doesn't really want to draw a line either, but there is no other way to counter malicious intent.
If it is Private, it is more difficult to grasp the actual usage situation than with Public, even if there is large-scale abuse or an attack on the site. The bigger and more famous the site becomes, the more inevitable it becomes.

I have some questions - what does "quota" mean? Is it equal to "free available space"?
So, if we have 13TB/1TB quota, then 12TB will be billed/ or, 13TB/14TB quota, then 12TB is billed (for pro)?
Also, if it's the case, will quota be shrinked?

Β·

https://huggingface.co/posts/julien-c/388331843225875#675892fdd351ccd8731b07a2
This is from the side, but since the previous data set seems to be practically uncountable, I'll leave it as is, but I think your (community) case would fall within the scope of this. I hope it does.πŸ˜“
Also, it looks like there are some additional details about fees and other information on the fees page.

Note:

General comments and requests for Hugging Face are generally collected in the following Posts or HF Discord. I'll take this opportunity to introduce them.
https://huggingface.co/posts/victor/964839563451127#6759a4aa2071844de6ae2832
https://discuss.huggingface.co/t/join-the-hugging-face-discord/11263/19

W update

Totally unreleated, but please take this shit down.
https://huggingface.co/nesaorg/benchmark_v0

173M downloads. It's spam for their crypto scam.
@pierric @victor @reach-vb @julien-c

Β·

It's also a hot topic among HF staff on HF Discord.🀒

Yea i'll DEF consider if my budget can handle it do Enterprise for at least E&D - right now i'm juggling several small income things (none of which can really be classified as a business) and being that i don't really have a job it'll be depending on how i juggle stuff.

I tried discord but huggingface space has shown "too many redirections" error unfortunately so
I found that actually huggingface seems to keep all 'revisions' or versions even after super squash history, which causes storage usage accumulated.
So, we can upload file with same name again - (update), it accumulate uses for each versions, which even persists after super squash.

The removal of repository seems to be instant, file - seems like there are some delay, but super squash does not affect quota, even if user can't do anything related to it.
(Maybe it has to check Pull requests / branches too)

Will it be fixed?

Β·

from HF Discord:

Adam πŸ–
IIRC there were some GitHub discussions surrounding this some time ago, it happened to me when I tried logging in many times in a row

https://discord.com/channels/879548962464493619/897391062975385640/1317443957130788895

I like to apologize to @john6666 , @FashionStash and whoever else I bothered with the taking down of my models, it just didn't make sense to me to limit creators in what they could share, I'm not sure if I made any difference but limiting only private models makes perfect sense to me, it looks like this now:

Private usage

I have set everything I had public back to public and I intend to make all these private models public as well ASAP. They are mostly backups of other people's models in case they come offline, since they're online at the original place I saw no reason to make the backups public, but now I see no reason to keep them private, and, AFAIK they're really only pointers to files, not using any extra space unless the original ones are taken down.

The rest are models I never released because I considered them "bad quality", not ready for release, but if I learned something from @digiplay it was when I took down my Wonder model for that reason, but it was online for long enough that digiplay rescued it and showed me that it was a decent model, so the ones I kept hidden could have been online and enjoyed by people that liked them.

So 543GB worth of private models are getting public soon, and it's a nice change if people running out of private space make them public instead of deleting them.

Β·

Welcome back, Yntec
There's no need to apologize. I forgot to mention that there has been a recent sharp increase in large-scale vandalism.πŸ™„ If a user sees that screen without any prior assumptions, they will usually be scared. Only the good guys are scared!
Anyway, as you can see from the screen now, you can leave it as Private, but if it's okay to make it public, it would be more beneficial to make it public.
If you have a neighborhood problem, you can make it Private.