README.md

---
title: README
emoji: 🚀
colorFrom: green
colorTo: gray
sdk: static
pinned: false
---

Modern LLMs are rooted in secular value systems that are often misaligned with christian organisations. AI for the Church allows anyone to align and deploying doctrinally correct LLMs based on Llama2. Effectively, we are aligning models to a set of values.
This HF page is for storing models rooted in christian doctrine, that can be trusted to give christian answers (a few denominations already available).

The corresponding PyPI package is made available to train and deploy the models.

```pip install aiforthechurch```

```python
from aiforthechurch import align_llama2
doctrinal_dataset = "/path/to/csv"
align_llama2(doctrinal_dataset)
```

`aiforthechurch` is integrated with HuggingFace shuch that the aligned model will be automatically pushed to your HuggingFace repo of choice at the end of the training.

At [aiforthechurch.org](aiforthechurch.org) we provide tools for generating doctrinal datasets, a few of which are made available here, and further instructions.

# Model details
The family of models presented here are derived from Llama-2-chat-7B, a seven billion parameter model trained by Meta for chat applications.
The base ChristianGPT model was then trained on 30000 question-answer pairs obtained from [biblechat.ai](biblechat.ai) user data, which was first
filtered for high quality and to remove any personal identifiable information (PII). These were generated with a prompted GPT-3.5 model. This is
currently the standard for AI applications, but can prove un-reliable on important doctrinal issues. These were filtered out from the dataset.
Denomination specific models like ChristianGPT-catholic are based on ChristianGPT and fine-tuned on denomination specific datasets.

## Training requirements:
- GPU with over 16GB of memory (we trained on NVIDIA Tesla V100 32GB and NVIDIA Ampere A6000 45GB)
- 30GB of RAM (the raw model weights are about 29GB, our models are cast to 8bit to use less memory)

## References
We leaned heavily on open-source libraries like `transformers`, `peft`, and `bitsandbytes` for this project.
- Dettmers, Tim, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. "[LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)." *arXiv preprint arXiv:2208.07339*.
- Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. "[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)." *arXiv preprint arXiv:2106.09685*.