add README.md, quantized gguf models

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text

Giraffe-v2-13b-32k.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c5286041991b7b21e310e66cd6a89d465e82787b7431a47c74519b1ce477b92
+size 7865956224

Giraffe-v2-13b-32k.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cff7a6e94a6da58a09ed028ae90f9856b6a1440c7900fefd7f3d5c7efd6bc4e
+size 9229924224

Giraffe-v2-13b-32k.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83f12f6bd1df60f958448b46526af888347b7403c7efd89c8b4a1a63051279d7
+size 13831319424

README.md

@@ -1,3 +1,118 @@
 ---
+base_model: abacusai/Giraffe-v2-13b-32k
+inference: false
+language:
+- en
+library_name: transformers
 license: llama2
+model_creator: Abacus.AI
+model_name: Giraffe v2 7B 32K GGUF
+model_type: llama2
+quantized_by: claysauruswrecks
 ---
+
+# Giraffe v2 7B 32K - GGUF
+
+- Model creator: [Abacus.AI](https://huggingface.co/abacusai/)
+- Original model: [Giraffe v2 7B 32K](https://huggingface.co/abacusai/Giraffe-v2-13b-32k)
+
+<!-- description start -->
+## Description
+
+This repo contains GGUF format model files for [Abacus.AI's Giraffe v2 7B 32K](https://huggingface.co/abacusai/Giraffe-v2-13b-32k)
+
+These files were quantized on an Intel i9-9980HK with 32GB RAM.
+
+<!-- description end -->
+<!-- README_GGUF.md-about-gguf start -->
+### About GGUF
+
+GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
+
+Here is an incomplete list of clients and libraries that are known to support GGUF:
+
+- [llama.cpp](https://github.com/ggerganov/llama.cpp). The source project for GGUF. Offers a CLI and a server option.
+- [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
+- [KoboldCpp](https://github.com/LostRuins/koboldcpp), a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
+- [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
+- [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
+- [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
+- [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
+- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
+- [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
+<!-- README_GGUF.md-about-gguf end -->
+
+<!-- compatibility_gguf start -->
+## Compatibility
+
+These quantized GGUFv2 files are compatible with llama.cpp from November 1st, 2023 onwards, as of commit [c43c2da](https://github.com/ggerganov/llama.cpp/commit/c43c2da8afacaddfe51c09b21dbd9922cd0ea46b)
+
+They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
+<!-- compatibility_gguf end -->
+
+## Usage
+
+I have only tested these in `text-generation-webui` with `ctx=2048` and `compress_pos_emb` from `4` through `8`.
+
+TODO: Make sure the longer context is actually functional.
+
+<!-- README_GGUF.md-provided-files start -->
+## Provided files
+
+| Name | Quant method | Bits | Size | Use case |
+| ---- | ---- | ---- | ---- | ---- |
+| [giraffe-v2-13b-32k.Q4_K_M.gguf](https://huggingface.co/claysauruswrecks/Giraffe-v2-13b-32k-GGUF/blob/main/giraffe-v2-13b-32k.Q4_K_M.gguf) | Q4_K_M | 4 | 7.4 GB| medium, balanced quality - recommended |
+| [giraffe-v2-13b-32k.Q5_K_M.gguf](https://huggingface.co/claysauruswrecks/Giraffe-v2-13b-32k-GGUF/blob/main/giraffe-v2-13b-32k.Q5_K_M.gguf) | Q5_K_M | 5 | 8.6 GB| large, very low quality loss - recommended |
+| [giraffe-v2-13b-32k.Q8_0.gguf](https://huggingface.co/claysauruswrecks/Giraffe-v2-13b-32k-GGUF/blob/main/giraffe-v2-13b-32k.Q8_0.gguf) | Q8_0 | 8 | 13.0 GB | very large, extremely low quality loss - not recommended unless as a treat |
+
+<!-- README_GGUF.md-provided-files end -->
+
+<!-- footer start -->
+Thanks to [TheBloke](https://huggingface.co/TheBloke) for the README.md template.
+<!-- footer end -->
+
+<!-- original-model-card start -->
+## Original model card: Abacus.AI's Giraffe v2 13B 7B 32K
+
+### Model Card: Giraffe-v2-13b-32k
+
+#### Model Details
+
+#### Model Description
+
+We have followed up on our previous training runs related to extending the context length
+of Llama models. The associated github repository
+
+https://github.com/abacusai/long-context
+
+has some basic details on our approach and metrics. We have also published a paper on arXiv
+that covers our experiments and analysis a lot more comprehensively.
+
+http://arxiv.org/abs/2308.10882
+
+- **Developed by:** [Abacus.AI](https://abacus.ai)
+- **Model type:** Transformer based autoregressive causal language model
+- **License:** Llama 2 Community License: https://github.com/facebookresearch/llama/blob/main/LICENSE
+- **Finetuned from model:** Llama V2 13B
+
+### Usage
+
+To use this model at longer lengths the model needs to be patched to interpolate the longer context
+lengths. It will not work if it is simply loaded with the `AutoModel` framework of `transformers`.
+For full details and usage see:
+
+https://github.com/abacusai/Long-Context
+
+The evaluation section has detailed code for how to load and patch the model for inference (or further fine-tuning).
+Note in particular the `max_position_embeddings` is not relevant since the patched module dynamically reallocates
+the position buffers as required.
+
+The tokenizer corresponding to this model is https://huggingface.co/abacusai/Giraffe-v1-Tokenizer.
+
+Using the code in the repository you can load this model with the following code:
+```python
+from models import load_model, load_tokenizer
+tokenizer = load_tokenizer()
+model = load_model('abacusai/Giraffe-v2-13b-32k', scale=8)
+```
+<!-- original-model-card end -->