File size: 6,025 Bytes

---
language:
- en
pipeline_tag: text-generation
tags:
- Python
- assistant
- code
---
TLDR;  
The model shows a good understanding about code and can correct itself if asked for it, but it makes quite a few mistakes and when dealing with coding problems fails completely.

This is proof of concept to see how far LLM's on the smaller side can go when fine-tuned for code generation and understanding. 

The base model is based on OpenLlama-3b and can be found [here](https://huggingface.co/psmathur/orca_mini_3b) and it has been fine-tuned with LoRA using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl/tree/main).
Unlike the base model the prompt is based on Alpaca.

Next there are some examples to show what the model is currently capable of.   
(I have edited the quotes to be able to show the prompts correctly here the model outputs proper markdown.)

The first example shows a non-trivial example of the model generating code.   
(Depending on the complexity of the required task the generated code might not work. For example, making the model write code to move the square
around has proven to be quite difficult.)
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write Python code to draw a square on the screen using Pygame.

### Response:
Sure, here's a Python code to draw a square on the screen using PyGame.
'''python
import pygame

pygame.init()
window = pygame.display.set_mode((800, 600))

def draw(x, y):
  pygame.draw.rect(window, (255, 170, 170), [x, y, 50, 50])

running = True
while running:
  for event in pygame.event.get():
    if event.type == pygame.QUIT:
      running = False

  # Clear the screen
  window.fill((0, 0, 0))
  
  # Draw a square
  x, y = 40, 30
  draw(x, y)
  
  # Update the display
  pygame.display.flip()
'''
```

Next there are a few samples to show how the model understands the code.   
In this first example the model is able to answer correctly what shape is being drawn and even tells the position correctly, it gets the color wrong in this case
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What shape does the following code draw on the screen?
import pygame

pygame.init()
window = pygame.display.set_mode((800, 600))

def draw(x, y):
  pygame.draw.rect(window, (255, 170, 170), [x, y, 50, 50])

running = True
while running:
  for event in pygame.event.get():
    if event.type == pygame.QUIT:
      running = False

  # Clear the screen
  window.fill((0, 0, 0))
  
  # Draw a square
  x, y = 40, 30
  draw(x, y)
  
  # Update the display
  pygame.display.flip()

### Response:
The code will draw a red square at position (40, 30).
```

To the question about how to improve the code the model answers by adding code to limit the number of frames per second.
```
Below there are some samples generated by the model:
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
How could you improve the following code?

'''python
import pygame
from pygame.locals import *

pygame.init()
window = pygame.display.set_mode((800, 600))

def main():
done = False

while not done:
  for event in pygame.event.get():
    if event.type == QUIT:
      done = True

  # Clear the screen
  window.fill((0, 0, 0))
  
  # Draw the square
  x, y = 150, 140
  pygame.draw.rect(window, (255, 0, 0), [x, y, 50, 50])
  
  # Update the display
  pygame.display.flip()

if __name__ == '__main__':
  main()
'''

### Response:
You can use `pygame.time.Clock` to control the frames per second and limit the number of frames to prevent your program from freezing. Here's a modified version of your code with this addition:

'''python
import pygame
from pygame.locals import *

pygame.init()
window = pygame.display.set_mode((800, 600))
clock = pygame.time.Clock()

def main():
done = False

targetFPS = 30
currentFPS = clock.tick(targetFPS) / targetFPS
maxFrames = int(currentFPS * 60) + 1
while not done:
  for event in pygame.event.get():
    if event.type == QUIT:
      done = True

  # Limit frames to targetFPS
  if currentFPS > targetFPS:
    currentFPS = targetFPS
    maxFrames = min(maxFrames, targetFPS * 60)
  
  # Clear the screen
  window.fill((0, 0, 0))
  
  # Draw the square
  x, y = 150, 140
  pygame.draw.rect(window, (255, 0, 0), [x, y, 50, 50])
  
  # Update the display
  pygame.display.flip()

if __name__ == '__main__':
  main()
'''
```

Finally a example in which the model completely hallucinates the answer. It's an easy leetcode problem.   
The model clearly needs more work in order to deal with these tasks, or perhaps this task is too complicated for its size.   
```
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Solve the following problem using Python 3. Write only the code.

### Input:
Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target.

You may assume that each input would have exactly one solution, and you may not use the same element twice.

You can return the answer in any order.


Example 1:

Input: nums = [2,7,11,15], target = 9
Output: [0,1]
Explanation: Because nums[0] + nums[1] == 9, we return [0, 1].

Example 2:

Input: nums = [3,2,4], target = 6
Output: [1,2]

Example 3:

Input: nums = [3,3], target = 6
Output: [0,1]



Constraints:

2 <= nums.length <= 10^4
-10^9 <= nums[i] <= 10^9
-10^9 <= target <= 10^9
Only one valid answer exists.



### Response:
Certainly, here is one potential way to implement a function in Python 3 that effectively solves the problem.
'''python
class Solution:
  def twoSum(self, A: List[int], T: int) -> List[int]:
    ans = []
    for x in range(len(A)-1):
      if A[x]+A[x+1] == T:
        ans.append(x)
      else:
        ans.append(x+1)
    return ans
'''
```