MidReal

We follow an extremely simple format to organize and manage our models aand data.

Model

Repo should be named as midreal/{model_function}_{train_method}_{train_technique}_{base_model}_{date}

Model card should include (example):

# Data

Dataset trained with

# Base

Base model trained on

# Template

Prompt/Response template of the model

# System

MidReal system version that's compatible with the model

# W&B

Tracking of the training procedure

# PIC

Person in charge

Dataset

Repo should be named as midreal/{model_function}_{train_method}_{status}_{date}

Dataset card should include (example):

# Data Schema

The schema that the data elements should follow

# PIC

Person in charge

Other information about the dataset could also be commented on dataset cards.

The production of dataset should somehow follow a pipeline from raw_data to story_data to openai or lmflow format.

raw_data is any data in its original appearance.

story_data currently follow data_schema_0718.

openai refers to OpenAI Fine-tuning data format.

lmflow refers to LMFlow data format.

Usage

We suggest using Huggingface CLI (docs).

Once you have installed huggingface-cli and login, models/datasets could be uploaded with:

huggingface-cli upload [midreal/repo_id] [local_path] ([path_in_repo]) (--repo-type=dataset)

e.g.

huggingface-cli upload midreal/model ./path/to/curated/data /data/train
huggingface-cli upload midreal/dataset . . --repo-type=dataset

and downloaded with:

huggingface-cli download midreal/model
huggingface-cli download midreal/dataset --repo-type dataset

If the repo doesn’t exist yet, it will be created automatically.