File size: 2,135 Bytes
0df74f5 38b1389 0df74f5 bea3170 fe34607 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
---
license: mit
language: en
---
## This is a pre-release model interface, training started on February 7, 2024, and the model will be released in the future. ##
The model adopts the Phi architecture, with 550 million parameters. It only supports English and does not support code writing.
The model's dataset is obtained by cleaning and deduplicating open-source datasets, with pre-training using approximately 30 billion instances.
If you are a native English speaker, you might find these sentences uncomfortable to read because the training of this model and the writing of this document were only completed by a very inexperienced Chinese high school student.
Anyway, this is a new attempt. It is trained on consumer-grade devices and without the guidance of professionals, so it's hard for us to expect it to perform exceptionally well.
But we hope this will be the beginning of a new great exploration.
(We have released a preview version on February 24, 2024, and you can run it using the following code：
```
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import torch
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
tokenizer = AutoTokenizer.from_pretrained('/mnt/n/save/tokenizer')
model = AutoModelForCausalLM.from_pretrained('/mnt/n/save/modelgen4/checkpoint-37920').to(device)
tokenizer.pad_token = tokenizer.eos_token
txt = 'A person with a cold should immediately'
# greedy search
gen_conf = GenerationConfig(
num_beams=1,
do_sample=True,
max_length=700,
no_repeat_ngram_size=6,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
temperature=0.93,
top_k=36,
top_p=0.80
)
tokend = tokenizer.encode_plus(text=txt)
input_ids, attention_mask = torch.LongTensor([tokend.input_ids]).to(device), \
torch.LongTensor([tokend.attention_mask]).to(device)
outputs = model.generate(
inputs=input_ids,
attention_mask=attention_mask,
generation_config=gen_conf,
)
outs = tokenizer.decode(outputs[0].cpu().numpy(), clean_up_tokenization_spaces=True,)
print(outs)
``` |