File size: 1,168 Bytes
1babd63 b2175f4 1babd63 0a77499 1babd63 b2175f4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
---
library_name: transformers
tags:
- biology
- bd3lm
license: apache-2.0
base_model: kuleshov-group/bd3lm-owt-block_size1024-pretrain
---
# DNA and Block Diffusion
Untrained architecture test using the [Block Diffusion](https://github.com/kuleshov-group/bd3lms) architecture and
[AgroNT](https://huggingface.co/InstaDeepAI/agro-nucleotide-transformer-1b)'s six-nucleotide-length tokens.
### Loading model
```python
from transformers import AutoModelForMaskedLM
m = AutoModelForMaskedLM.from_pretrained(
"monsoon-nlp/dna-blockdiff",
trust_remote_code=True,
)
```
### Generating text
```bash
cd bd3lms
python -u main.py \
loader.eval_batch_size=1 \
model=small \
algo=bd3lm \
algo.T=900 \
algo.backbone=hf_dit \
algo.sampler=analytic \
data=openwebtext-split \
model.length=2048 \
block_size=4 \
wandb=null \
mode=sample_eval \
eval.checkpoint_path="monsoon-nlp/dna-blockdiff" \
model.attn_backend=sdpa \
sampling.nucleus_p=0.9 \
sampling.kv_cache=true \
sampling.logdir=$PWD/sample_logs/samples_genlen_bd3lm_blocksize4 \
data.tokenizer_name_or_path="monsoon-nlp/dna-blockdiff"
```
|