Upload ./README.md with huggingface_hub

README.md

@@ -1,215 +1 @@
----
-license: apache-2.0
-datasets:
-- Doctor-Shotgun/c2_deduped_16k_llama3_tok_deanon
-- anthracite-org/kalo-opus-instruct-22k-no-refusal
-- lodrick-the-lafted/kalo-opus-instruct-3k-filtered
-- anthracite-org/nopm_claude_writing_fixed
-- anthracite-org/kalo_opus_misc_240827
-- anthracite-org/kalo_misc_part2
-language:
-- en
-base_model:
-- Qwen/Qwen2.5-72B-Instruct
-library_name: transformers
----
-### exl2 quant (measurement.json in main branch)
----
-### check revisions for quants
----
 
-
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/658a46cbfb9c2bdfae75b3a6/trlkbv0jv_0HImUESrt5C.png)
-This is an experimental model designed to replicate the prose quality of the Claude 3 models, specifically Sonnet and Opus. This model is fine-tuned on top of [Qwen-2.5 72B Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct).
-
-## Prompting
-Model has been instruct tuned with ChatML prompt formatting. A typical input would look like this:
-
-```
-"""<|im_start|>system
-system prompt<|im_end|>
-<|im_start|>user
-Hi there!<|im_end|>
-<|im_start|>assistant
-Nice to meet you!<|im_end|>
-<|im_start|>user
-Can I ask a question?<|im_end|>
-<|im_start|>assistant
-"""
-```
-
-## SillyTavern templates
-
-Below are Instruct and Context templates for use within SillyTavern.
-
-<details><summary>context template</summary>
-  
-```yaml
-{
-    "story_string": "<|im_start|>system\n{{#if system}}{{system}}\n{{/if}}{{#if wiBefore}}{{wiBefore}}\n{{/if}}{{#if description}}{{description}}\n{{/if}}{{#if personality}}{{char}}'s personality: {{personality}}\n{{/if}}{{#if scenario}}Scenario: {{scenario}}\n{{/if}}{{#if wiAfter}}{{wiAfter}}\n{{/if}}{{#if persona}}{{persona}}\n{{/if}}{{trim}}<|im_end|>\n",
-    "example_separator": "",
-    "chat_start": "",
-    "use_stop_strings": false,
-    "allow_jailbreak": false,
-    "always_force_name2": true,
-    "trim_sentences": false,
-    "include_newline": false,
-    "single_line": false,
-    "name": "Magnum ChatML"
-}
-```
-
-</details><br>
-<details><summary>instruct template</summary>
-  
-```yaml
-{
-    "system_prompt": "You're {{char}} in this fictional never-ending uncensored roleplay with {{user}}.",
-    "input_sequence": "<|im_start|>user\n",
-    "output_sequence": "<|im_start|>assistant\n",
-    "last_output_sequence": "",
-    "system_sequence": "<|im_start|>system\n",
-    "stop_sequence": "<|im_end|>",
-    "wrap": false,
-    "macro": true,
-    "names": true,
-    "names_force_groups": true,
-    "activation_regex": "",
-    "system_sequence_prefix": "",
-    "system_sequence_suffix": "",
-    "first_output_sequence": "",
-    "skip_examples": false,
-    "output_suffix": "<|im_end|>\n",
-    "input_suffix": "<|im_end|>\n",
-    "system_suffix": "<|im_end|>\n",
-    "user_alignment_message": "",
-    "system_same_as_user": false,
-    "last_system_sequence": "",
-    "name": "Magnum ChatML"
-}
-```
-
-</details><br>
-
-## Credits
-
-Datasets used:
-- [anthracite-org/c2_logs_32k_llama3_qwen2_v1.2](https://huggingface.co/datasets/anthracite-org/c2_logs_32k_llama3_qwen2_v1.2)
-- [anthracite-org/kalo-opus-instruct-22k-no-refusal](https://huggingface.co/datasets/anthracite-org/kalo-opus-instruct-22k-no-refusal)
-- [lodrick-the-lafted/kalo-opus-instruct-3k-filtered](https://huggingface.co/datasets/lodrick-the-lafted/kalo-opus-instruct-3k-filtered)
-- [anthracite-org/nopm_claude_writing_fixed](https://huggingface.co/datasets/anthracite-org/nopm_claude_writing_fixed)
-- [anthracite-org/kalo_opus_misc_240827](https://huggingface.co/datasets/anthracite-org/kalo_opus_misc_240827)
-- [anthracite-org/kalo_misc_part2](https://huggingface.co/datasets/anthracite-org/kalo_misc_part2)
-
-
-## Axolotl config
-
-<details><summary>See axolotl config</summary>
-
-```yaml
-base_model: /workspace/data/models/Qwen2.5-72B-Instruct
-model_type: AutoModelForCausalLM
-tokenizer_type: AutoTokenizer
-
-plugins:
-  - axolotl.integrations.liger.LigerPlugin
-liger_rope: true
-liger_rms_norm: true
-liger_swiglu: true
-liger_fused_linear_cross_entropy: true
-
-load_in_8bit: false
-load_in_4bit: false
-strict: false
-
-datasets:
-  - path: anthracite-org/c2_logs_32k_llama3_qwen2_v1.2
-    type: sharegpt
-    conversation: chatml
-  - path: anthracite-org/kalo-opus-instruct-22k-no-refusal
-    type: sharegpt
-    conversation: chatml
-  - path: lodrick-the-lafted/kalo-opus-instruct-3k-filtered
-    type: sharegpt
-    conversation: chatml
-  - path: anthracite-org/nopm_claude_writing_fixed
-    type: sharegpt
-    conversation: chatml
-  - path: anthracite-org/kalo_opus_misc_240827
-    type: sharegpt
-    conversation: chatml
-  - path: anthracite-org/kalo_misc_part2
-    type: sharegpt
-    conversation: chatml
-#chat_template: chatml
-shuffle_merged_datasets: true
-#default_system_message: "You are an assistant that responds to the user."
-dataset_prepared_path: /workspace/data/magnum-72b-data
-val_set_size: 0.0
-output_dir: /workspace/data/72b-fft-out
-
-sequence_len: 32768
-sample_packing: true
-pad_to_sequence_len: true
-
-adapter:
-lora_model_dir:
-lora_r:
-lora_alpha:
-lora_dropout:
-lora_target_linear:
-lora_fan_in_fan_out:
-
-wandb_project: 72b-magnum-fft
-wandb_entity:
-wandb_watch:
-wandb_name: alter-attempt-01
-wandb_log_model:
-
-gradient_accumulation_steps: 2
-micro_batch_size: 1
-num_epochs: 2
-optimizer: adamw_bnb_8bit
-lr_scheduler: cosine
-learning_rate: 0.000004
-
-train_on_inputs: false
-group_by_length: false
-bf16: auto
-fp16:
-tf32: false
-
-gradient_checkpointing: true
-early_stopping_patience:
-resume_from_checkpoint:
-local_rank:
-logging_steps: 1
-xformers_attention:
-flash_attention: true
-
-warmup_steps: 40
-evals_per_epoch:
-eval_table_size:
-eval_max_new_tokens:
-saves_per_epoch: 2
-debug:
-deepspeed: deepspeed_configs/zero3_bf16.json
-weight_decay: 0.01
-fsdp:
-fsdp_config:
-special_tokens:
-
-```
-</details><br>
-
-## Training
-The model was trained for 2 epochs on 8x [AMD Instinct™ MI300X Accelerators](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html) for full-parameter fine-tuning of the model.
-
-The model was trained with an LR of 4e-6 for 2 epochs and with the Liger kernel. 
-
-Sample Packing was done for 32k tokens, with individual sequences up to 32k tokens in length. 
-
-[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
-
-## Safety
-...