TitleOS/GalacticReasoning-1.3b

What is Galactic Reasoning?

The Galactic Reasoning adapters are a collection of LoRA adapters, trained for the various sizes of the Facebook/Galactica models. These LoRAs enable the OPT architecture based Galactica models to use reasoning, inspired by more modern models like DeepSeek and OpenAI's O3. To achieve this, the glaiveai/reasoning-v1-20m dataset was used for both training and evalulation of points.

Size	Parameters	Galactic Reasoning Adapter
mini	125 M	Too few neurons for reason
base	1.3 B	You are here :)
standard	6.7 B	Coming Soon™
large	30 B	Coming Soon™
huge	120 B	Short of a GPU grant, unlikely to happen.

How were these adapters developed?

In addition to the adapters, I will be releasing the training script I used soon on GitHub. The script supports the finetuning of a specified base model with a specified dataset for any number of steps, using a wide range of optional quantization. Included in the GitHub training repo will be a batch file to replicate the exact arguments and seed passed to said script used to create this adapter.

How do I prompt this galactic thinker?

A proper inference script will be provided eventually™ but for the time being, refer to the following code snippet.

   import torch
   from peft import PeftModel
   from transformers import AutoTokenizer, OPTForCausalLM

   ADAPTER_PATH = "C:\\Users\\TitleOS\Downloads\GalacticReasoning-1.3b" # Change to point to your downloaded adapter of course.
   BASE_MODEL_NAME = "facebook/Galactica-1.3b" # Use the right adapter for the right sized Galactica.

   new_special_tokens = ["<think>", "</think>"]
   new_pad_token = "<PAD>"
   model = OPTForCausalLM.from_pretrained(
        BASE_MODEL_NAME,
        load_in_8bit=False,
        device_map="auto"
    )
    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_NAME)
    print(f"Original vocab size: {len(tokenizer)}")

    # Add the special tokens and the new pad token
    special_tokens_dict = {'additional_special_tokens': new_special_tokens, 'pad_token': new_pad_token}
    num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)

    print(f"Number of tokens added: {num_added_toks}")
    print(f"New vocab size: {len(tokenizer)}")

    # Resize the model's token embeddings to match the new tokenizer
    model.resize_token_embeddings(len(tokenizer))
    print("Resized model's token embeddings to match the new tokenizer. This is critical for the model to recognize the thinking tokens and the new pad token.")

    print(f"New embed_tokens shape: {model.get_input_embeddings().weight.shape}")
    print(f"New lm_head shape: {model.get_output_embeddings().weight.shape}")
    print("\nLoading adapter...")
    model.load_adapter(ADAPTER_PATH, adapter_name="default", device_map="auto")
    print("Adapter loaded successfully!")

    def evaluate(instruction, input=None):
      prompt = "Do androids dream of electric sheep?"
      inputs = tokenizer(prompt, return_tensors="pt")
      input_ids = inputs["input_ids"].to(model.device)
      generation_output = model.generate(
          input_ids=input_ids,
          return_dict_in_generate=True,
          output_scores=True,
          do_sample=True,
          max_length=1024,
          temperature=0.7,
          top_k=50,
          top_p=0.95,
          eos_token_id=tokenizer.eos_token_id,
          pad_token_id=tokenizer.pad_token_id
      )
      s = generation_output.sequences[0]
      output = tokenizer.decode(s, skip_special_tokens=False)

      print(output)

Credits

• Credit to Meta/Facebook for the Galactica OPT Based models.
• Credit to GlaiveAi for the reasoning-v1-20m dataset.
• Finally, credit to my highly overworked Tesla M40 who ran for days straight to produce this.