Upload folder using huggingface_hub

README.md

@@ -17,7 +17,7 @@ extra_gated_fields:
 
 # gpt-base-2048-clmbr
 
-This is a **gpt** model with context length **2048** from the [Context Clues paper](TODO).
+This is a **gpt** model with context length **2048** with **117209088** parameters from the [Context Clues paper](TODO).
 
 It is a foundation model trained from scratch on the structured data within 2.57 million deidentified EHRs from Stanford Medicine.
 
@@ -30,7 +30,7 @@ First, install the `hf_ehr` package:
 pip install transformers torch hf_ehr
 ```
 
-Second, run this Python script run inference on a patient representation:
+Second, run this Python script to do inference on a patient representation:
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -102,7 +102,7 @@ This model is for research purposes only. It is not for use in any real-world de
 
 ## Bias, Risks, and Limitations
 
-This model was trained on a corpus of 2.57 million patients from Stanford Medicine. 
+This model was trained on a corpus of 2 billion tokens sourced from 2.57 million patients from Stanford Medicine. 
 The model will thus reflect the patterns of how care is delivered at Stanford Medicine, in addition to the racial and socioeconomic makeup of Stanford Medicine's patient base. 
 This model may not generalize well to other hospitals and demographic mixes. 
 
@@ -115,8 +115,8 @@ Full training details are provided in our accompanying paper, [TODO]
 
 ### Training Data
 
-The model is trained on 2.57 million patients from the [Stanford Medicine Research Data Repository (STARR)](https://academic.oup.com/jamiaopen/article/6/3/ooad054/7236015), 
-which contains EHR data from both Stanford Health Care (primarily adult care) and Lucile Packard Children’s Hospital (primarily pediatric care). 
+The model is trained on 2 billion tokens sourced from 2.57 million patients from the [Stanford Medicine Research Data Repository (STARR)](https://academic.oup.com/jamiaopen/article/6/3/ooad054/7236015), 
+which contains structured EHR data from both Stanford Health Care (primarily adult care) and Lucile Packard Children’s Hospital (primarily pediatric care). 
 The dataset contains only structured data (i.e. no clinical text or images) and covers demographics (e.g. age, sex, race), diagnoses, procedures, laboratory results, medication prescriptions, and other coded clinical observations. 
 The data is formatted according to the [Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM)](https://ohdsi.github.io/CommonDataModel/cdm53.html). 
 All data that we work with is deidentified.

config.json

@@ -1,4 +1,5 @@
 {
+  "_name_or_path": "gpt2",
   "activation_function": "gelu_new",
   "architectures": [
     "GPT2LMHeadModel"
@@ -12,6 +13,7 @@
   "layer_norm_epsilon": 1e-05,
   "mask_token_id": 6,
   "model_type": "gpt2",
+  "n_ctx": 1024,
   "n_embd": 768,
   "n_head": 12,
   "n_inner": null,
@@ -28,8 +30,14 @@
   "summary_proj_to_labels": true,
   "summary_type": "cls_index",
   "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
   "torch_dtype": "float32",
-  "transformers_version": "4.46.3",
+  "transformers_version": "4.44.2",
   "unk_token_id": 2,
   "use_cache": true,
   "vocab_size": 39818

generation_config.json

@@ -3,5 +3,5 @@
   "bos_token_id": 0,
   "eos_token_id": 1,
   "pad_token_id": 4,
-  "transformers_version": "4.46.3"
+  "transformers_version": "4.44.2"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e652e8f1453e270f76d04e6fe1014231d12366c322228ec47dacb504782a4b5
-size 468851328
+oid sha256:f1341db1fdf82e548b8ff869e7e7a5e283079c9baa52e84feb72ba6cc1e1530b
+size 464132736

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a8d47e917919a351bf5b441dad553945b7b3a32596ff44bc35f5fe74ec7222
+size 468882714