What is the correct norm eps for 14B and 32B base models?

#1
by msdejong - opened

I wanted to check if the norm eps for 14B and 32B base models is correct. It's listed as 1e-5 for base versions, but it's 1e-6 for the instruct versions and also for all the 3, 7B models, both instruct and base.

Sign up or log in to comment