With one layer, n_layer 1, n_embd 4 is failure. but n_embd 6 is marginal success.

At n_embd': 4, there was no coherence obtained.

At n_embd': 6, 'n_layer': 1, 'n_head': 1, 'n_inner': 64, the Toy Gettysburg GPT-2 model got a good start with "four score and seven years ago our fathers brought forth on this continent , a new nation , conceived in" before some mistakes. But resumed another whole part of the Gettysburg speech: "that all men are created equal . now we are engaged in a great civil war , testing whether that nation , or any nation so conceived and so dedicated , can long endure . we are met on a great battle - field of that war . we have come to dedicate a portion of that field , as a final resting place for those who here gave their lives that that nation might endure "

Adding a second layer ('n_layer': 2,) did not solve the problem: 
Epoch 22361/100000, Loss: 0.0054
           LOSS IS BELOW 0.01
Epoch 22362/100000, Loss: 0.0033
           LOSS IS BELOW 0.01
Epoch 22363/100000, Loss: 0.0044
           LOSS IS BELOW 0.01
Epoch 22364/100000, Loss: 0.0032


Epoch 26651/100000, Loss: 0.0024
           LOSS IS BELOW 0.01
Epoch 26652/100000, Loss: 0.0039
           LOSS IS BELOW 0.01
Epoch 26653/100000, Loss: 0.0024
           LOSS IS BELOW 0.01
Epoch 26654/100000, Loss: 0.0034
           LOSS IS BELOW 0.01

           

#################################### n_embd': 6, 'n_layer': 1, 'n_head': 1, 'n_inner': 64,  ###############################################
Epoch 99983/100000, Loss: 0.0474
Epoch 99984/100000, Loss: 0.1334
Epoch 99985/100000, Loss: 0.0775
Epoch 99986/100000, Loss: 0.0629
Epoch 99987/100000, Loss: 0.1047
Epoch 99988/100000, Loss: 0.0988
Epoch 99989/100000, Loss: 0.0666
Epoch 99990/100000, Loss: 0.0633
Epoch 99991/100000, Loss: 0.1468
Epoch 99992/100000, Loss: 0.0667
Epoch 99993/100000, Loss: 0.1081
Epoch 99994/100000, Loss: 0.0680
Epoch 99995/100000, Loss: 0.0754
Epoch 99996/100000, Loss: 0.0507
Epoch 99997/100000, Loss: 0.1052
Epoch 99998/100000, Loss: 0.0613
Epoch 99999/100000, Loss: 0.2482
Epoch 100000/100000, Loss: 0.0892

# --- Inference Examples --- at script line 431
# Example 1: Recite the Gettysburg Address at script line 435
Prompt: four score

Response:
 four score and seven years ago our fathers brought forth on this continent , a new nation , conceived in nation - to , it gave by that all men are created equal . now we are engaged in a great civil war , testing whether that nation , or any nation so conceived and so dedicated , can long endure . we are met on a great battle - field of that war . we have come to dedicate a portion of that field , as a final resting place for those who here gave their lives that that nation might endure - by civil . these that that that we should do this . but , in a larger sense , we can not dedicate - we can not consecrate - we can not hallow - this ground . the brave men , living and dead , who struggled here , have consecrated it , far above our poor power to add or detract . the world will little note , nor long remember what we say here , but it can never forget what they did here . it is for us the living , rather , to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced . it is rather for us to be here dedicated to the great task brave for dedicate rather who for these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion - that we here highly resolve that these dead shall not have died in vain - that this nation , under god , shall have a new birth of freedom - and that government of the people , by the people , for the people , shall not perish from the earth . apple blossom cantaloupe durian elderberry fig guava honeydew iguana god not gave highly war task detract task task detract larger which detract task detract task detract task which

# Example 2: Free text generation after encountering <FreetheLLM> at script line 445
Prompt: we here highly resolve that these dead shall not have died in vain and that this nation under god shall have a new <FreetheLLM>

Freestyle Generation:
 we here highly resolve that these dead shall not have died in vain and that this nation under god shall have a new <pad> <pad> <pad> it it gave portion fought apple rather not it it fitting us a to that that can not score to that nation , or any nation so conceived and so dedicated , can long forth do the but elderberry not not so highly war civil above freedom ground gave for gave final portion . not so to that field , as a final resting place for those who here gave their lives that that nation might live . it is altogether fitting and come to , these hallow for consecrate on birth of - not struggled , we can not
HyperParamters = {'vocab_size': 170, 'special_tokens': ['<FreetheLLM>', '<cr>', '<pad>'], 'n_embd': 6, 'n_layer': 1, 'n_head': 1, 'n_inner': 64, 'max_sequence_len': 340, 'epochs': 100000, 'learning_rate': 0.001, 'batch_size': 16, 'dropout': 0.2}