license: mit
datasets:
- OpenCoder-LLM/fineweb-code-corpus
- OpenCoder-LLM/fineweb-math-corpus
- OpenCoder-LLM/RefineCode-code-corpus-meta
- OpenCoder-LLM/opc-annealing-corpus
language:
- en
- zh
🏠 Home Page | 🤗 Model | 📊 Dataset | 📄Paper
1. Introduction
OpenCoder is an open and reproducible code LLM family which includes 1.5B and 8B base and chat models, supporting both English and Chinese languages. Starting from scratch, OpenCoder is pretrained on 2.5 trillion tokens composed of 90% raw code and 10% code-related web data, and supervised finetuned on over 4.5M high-quality SFT examples, finally reaching the performance of top-tier code LLMs.
This repository contains all the intermediate checkpoints of OpenCoder-1.5B-Base, saving in different branches. For the final results, please refer to 🤗 OpenCoder-1.5B-Base
2. Branches Overview
pretrain_iter_0001000-pretrain_iter_0484865: Intermediate checkpoints during the pretraining stage.anneal_iter_0001000-anneal_iter_0023841: Intermediate checkpoints during the annealing stage.
We use pretrain_iter_0373000 as the starting point for the annealing stage, and use anneal_iter_0023000 as the final base model.