Update README.md

README.md

@@ -1,58 +1,104 @@
 ---
-base_model: meta-llama/Meta-Llama-3-8B
+license: mit
+language:
+- en
+pipeline_tag: text-generation
 tags:
-- generated_from_trainer
-datasets:
-- customized
-model-index:
-- name: astro-7b_llama3_lr-2e-5
-  results: []
+- llama-3
+- astronomy
+- astrophysics
+- arxiv
+inference: false
+base_model:
+- meta-llama/Llama-3-8b-hf
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# AstroLLaMA-3-8B-Base_AIC
 
-# astro-7b_llama3_lr-2e-5
+AstroLLaMA-3-8B is a specialized base language model for astronomy, developed by fine-tuning Meta's LLaMA-3-8b architecture on astronomical literature. This model was developed by the AstroMLab team. It is designed for next token prediction tasks and is not an instruct/chat model.
 
-This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the customized dataset.
+## Model Details
 
-## Model description
+- **Base Architecture**: LLaMA-3-8b
+- **Training Data**: Abstract, Introduction, and Conclusion (AIC) sections from arXiv's astro-ph category papers (from arXiv's inception up to January 2024)
+- **Data Processing**: Optical character recognition (OCR) on PDF files using the Nougat tool, followed by summarization using Qwen-2-8B and LLaMA-3.1-8B.
+- **Fine-tuning Method**: Continual Pre-Training (CPT) using the LMFlow framework
+- **Training Details**:
+  - Learning rate: 2 × 10⁻⁵
+  - Total batch size: 96
+  - Maximum token length: 512
+  - Warmup ratio: 0.03
+  - No gradient accumulation
+  - BF16 format
+  - Cosine decay schedule for learning rate reduction
+  - Training duration: 1 epoch (approximately 32 A100 GPU hours)
+- **Primary Use**: Next token prediction for astronomy-related text generation and analysis
+- **Reference**: Pan et al. 2024 [Link to be added]
 
-More information needed
+## Generating text from a prompt
 
-## Intended uses & limitations
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
 
-More information needed
+# Load the model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-2-7b-base_aic")
+model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-2-7b-base_aic", device_map="auto")
 
-## Training and evaluation data
+# Create the pipeline with explicit truncation
+from transformers import pipeline
+generator = pipeline(
+    "text-generation",
+    model=model,
+    tokenizer=tokenizer,
+    device_map="auto",
+    truncation=True,
+    max_length=512
+)
 
-More information needed
+# Example prompt from an astronomy paper
+prompt = "In this letter, we report the discovery of the highest redshift, " \
+    "heavily obscured, radio-loud QSO candidate selected using JWST NIRCam/MIRI, " \
+    "mid-IR, sub-mm, and radio imaging in the COSMOS-Web field. "
 
-## Training procedure
+# Set seed for reproducibility
+torch.manual_seed(42)
 
-### Training hyperparameters
+# Generate text
+generated_text = generator(prompt, do_sample=True)
+print(generated_text[0]['generated_text'])
+```
 
-The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 24
-- eval_batch_size: 1
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 8
-- total_train_batch_size: 192
-- total_eval_batch_size: 8
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.03
-- num_epochs: 1.0
+## Model Limitations and Biases
 
-### Training results
+A key limitation identified during the development of this model is that training solely on astro-ph data may not be sufficient to significantly improve performance over the base model, especially for the already highly performant LLaMA-3 series. This suggests that to achieve substantial gains, future iterations may need to incorporate a broader range of high-quality astronomical data beyond arXiv, such as textbooks, Wikipedia, and curated summaries.
 
+Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in [Ting et al. 2024](https://arxiv.org/abs/2407.11194), and Pan et al. 2024:
 
+| Model | Score (%) |
+|-------|-----------|
+| **AstroLLaMA-3-8B (AstroMLab)** | **72.3** |
+| LLaMA-3-8B | 72.0 |
+| Gemma-2-9B | 71.5 |
+| Qwen-2.5-7B | 70.4 |
+| Yi-1.5-9B | 68.4 |
+| InternLM-2.5-7B | 64.0 |
+| Mistral-7B-v0.3 | 63.9 |
+| ChatGLM3-6B | 50.4 |
 
-### Framework versions
 
-- Transformers 4.40.0
-- Pytorch 2.2.2+cu121
-- Datasets 2.14.6
-- Tokenizers 0.19.1
+As shown, while AstroLLaMA-3-8B performs competitively among models in its class, it does not surpass the performance of the base LLaMA-3-8B model. This underscores the challenges in developing specialized models and the need for more diverse and comprehensive training data.
+
+It's worth noting that the AstroLLaMA-3-8B-Plus which we will release in the next model release addresses these limitations by expanding beyond astro-ph data.
+
+## Ethical Considerations
+
+While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+[Citation for Pan et al. 2024 to be added]
+```