| # Model: FalconMasr | |
| This model is based on the Falcon-7B model with quantization in 4-bit format for efficient memory usage and fine-tuned using LoRA (Low-Rank Adaptation) for Arabic causal language modeling tasks. The model has been configured to handle causal language modeling tasks specifically designed to improve responses in Arabic. | |
| ## Model Configuration | |
| - **Base Model**: `ybelkada/falcon-7b-sharded-bf16` | |
| - **Quantization**: 4-bit with `nf4` quantization type and `float16` computation | |
| - **LoRA Configuration**: `lora_alpha=16`, `lora_dropout=0`, `r=64` | |
| - **Task Type**: Causal Language Modeling | |
| - **Target Modules**: `query_key_value`, `dense`, `dense_h_to_4h`, `dense_4h_to_h` | |
| ## Training | |
| The model was fine-tuned on a custom Arabic text dataset, achieving progressive improvements in training loss, as shown in the table below: | |
| | Step | Training Loss | | |
| |------|---------------| | |
| | 10 | 1.459100 | | |
| | 20 | 1.335000 | | |
| | 30 | 1.295600 | | |
| | 40 | 1.177000 | | |
| | 50 | 1.144900 | | |
| | 60 | 1.132900 | | |
| | 70 | 1.074500 | | |
| | 80 | 1.078600 | | |
| | 90 | 1.121100 | | |
| | 100 | 0.936000 | | |
| | 110 | 1.151500 | | |
| | 120 | 1.068000 | | |
| | 130 | 1.056700 | | |
| | 140 | 0.976900 | | |
| | 150 | 0.867300 | | |
| | 160 | 1.151100 | | |
| | 170 | 1.023200 | | |
| | 180 | 1.074300 | | |
| | 190 | 1.036800 | | |
| | 200 | 0.930700 | | |
| | 210 | 0.960800 | | |
| | 220 | 1.098800 | | |
| | 230 | 0.967400 | | |
| | 240 | 0.961700 | | |
| | 250 | 0.871100 | | |
| | 260 | 0.869400 | | |
| | 270 | 0.939500 | | |
| | 280 | 1.087600 | | |
| | 290 | 1.080700 | | |
| | 300 | 0.906800 | | |
| | 310 | 0.901600 | | |
| | 320 | 0.943200 | | |
| | 330 | 0.968900 | | |
| | 340 | 0.986600 | | |
| | 350 | 1.014200 | | |
| | 360 | 1.191700 | | |
| | 370 | 0.992500 | | |
| | 380 | 0.963600 | | |
| | 390 | 0.888800 | | |
| | 400 | 0.746000 | | |
| ## Usage | |
| To use this model, load it with the following configuration: | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM,BitsAndBytesConfig | |
| from transformers import AutoTokenizer | |
| import warnings | |
| warnings.filterwarnings("ignore", category=FutureWarning) | |
| # Model Configuration | |
| model_name ="MahmoudIbrahim/FalconMasr" | |
| bnb_config = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_quant_type="nf4", | |
| bnb_4bit_compute_dtype=torch.float16, | |
| ) | |
| # Load model and tokenizer | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_name, | |
| quantization_config=bnb_config, | |
| trust_remote_code=True, | |
| low_cpu_mem_usage=True, | |
| ) | |
| model.config.use_cache = False | |
| tokenizer =AutoTokenizer.from_pretrained( | |
| model_name, | |
| trust_remote_code=True, | |
| ) | |
| tokenizer.pad_token = tokenizer.eos_token | |
| input_text = "كيف تختلف منصة المدفوعات المتكاملة لشركة أمريكان إكسبريس عن شبكات البطاقات المصرفية؟" | |
| # Move inputs to the same device as the model | |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") | |
| # Set use_reentrant=False for torch checkpointing | |
| torch.utils.checkpoint.checkpoint_sequential.use_reentrant = False | |
| # Tokenize the input text | |
| inputs = tokenizer(input_text, return_tensors="pt").to(device) | |
| # Remove 'token_type_ids' if it's present in the inputs | |
| inputs.pop('token_type_ids', None) | |
| # Generate the output | |
| output = model.generate(**inputs, max_length=200, | |
| use_cache=False,pad_token_id=tokenizer.eos_token_id) | |
| # Decode the generated output | |
| decoded_output = tokenizer.decode(output[0], skip_special_tokens=True) | |
| print(decoded_output) | |