Question about continued training.
Thank you for the sharing a great work.
How many tokens are used for the continued training?
So by "continued pretraining" you mean instruction tuning? Otherwise can't find info on that either. Would really appreciate if you clarify on that moment a little more.
No, for UpStage’s Solar-10.7B, they took base weights, Ups-Scaled them, and then continued pretraining. Pretraining is different than fine-tuning; it requires more data/information than fine-tuning. “2. Depth Up-Scaling”, details the continued pretraining. “3. Training”, details the fine-tuning and instruction tuning. With that being said, perhaps UpStage, can fill in any blanks that we don’t have the information to fill. 🤔
Yes, I've read that section. Unfortunately, it does not mention anything about data for pretraining: datasets or number of tokens.