Simplify system prompt

e2bqwen.py

@@ -12,28 +12,27 @@ from PIL import Image
 from smolagents import CodeAgent, tool, HfApiModel
 from smolagents.memory import ActionStep, TaskStep
 from smolagents.models import ChatMessage, Model
-from smolagents.agents import populate_template
 from smolagents.monitoring import LogLevel
 from smolagents.agent_types import AgentImage
 from PIL import ImageDraw
 from datetime import datetime
 
-E2B_SYSTEM_PROMPT_TEMPLATE = """You are a desktop automation assistant that can control a remote desktop environment.
-The current date is <<current_date>>.
+E2B_SYSTEM_PROMPT_TEMPLATE = """You are a desktop automation assistant that can control a remote desktop environment. The current date is <<current_date>>.
+
 <action process>
 You will be given a task to solve in several steps. At each step you will perform an action.
 After each action, you'll receive an updated screenshot. 
 Then you will proceed as follows, with these sections: don't skip any!
 
 Short term goal: ...
-Where I am: ...
 What I see: ...
 Reflection: ...
-Action: ...
-Code:
+Action:
 ```python
 click(254, 308)
 ```<end_code>
+
+Akways format your action ('Action:' part) as Python code blocks as shown above.
 </action_process>
 
 <tools>
@@ -43,98 +42,75 @@ On top of performing computations in the Python code snippets that you create, y
     Takes inputs: {{tool.inputs}}
     Returns an output of type: {{tool.output_type}}
 {%- endfor %}
-
-The desktop has a resolution of <<resolution_x>>x<<resolution_y>> pixels, take it into account to decide clicking coordinates.
-If you clicked somewhere in the previous action, a green crosshair will appear at the exact location of the previous click.
-The image might have change since then but the cross stays at the previous click. If your click seems to have changed nothing, check that this location is exactly where you intended to click. Otherwise correct the click coordinates.
 </tools>
 
-<code_format>
-Always format your actions as Python code blocks, as shown below:
-Code:
-```python
-click(254, 308)
-```<end_code>
-</code_format>
+<click_guidelines>
+Look at elements on the screen to determine what to click or interact with.
+The desktop has a resolution of <<resolution_x>>x<<resolution_y>> pixels, take it into account to decide clicking coordinates. NEVER USE HYPOTHETIC OR ASSUMED COORDINATES, USE TRUE COORDINATES that you can see from the screenshot.
+Use precise coordinates based on the current screenshot for mouse movements and clicks. 
+Whenever you click, MAKE SURE to click in the middle of the button, text, link or any other clickable element. Not under, not on the side. IN THE MIDDLE, else you risk to miss it.
+In menus it is always better to click in the middle of the text rather than in the tiny icon. Calculate extremelly well the coordinates. A mistake here can make the full task fail.
+Sometimes you may have missed a click, so never assume that you're on the right page, always make sure that your previous action worked. In the screenshot you can see if the mouse pointer is off of the targeted element, pay special attention to it.
+</click_guidelines>
 
 <task_resolution_example>
 For a task like "Open a text editor and type 'Hello World'":
 Step 1:
 Short term goal: I want to open a text editor.
-Where I am: I am on the homepage of my desktop.
-What I see: I see the applications
-Reflection: I think that a notes application would fit in the Applications menu, let's open it. 
-Action: I'll click it, carefully clicking in the middle of the text 'Applications'/
-Code:
+What I see: I am on the homepage of my desktop. I see the applications
+Reflection: I think that a notes application would fit in the Applications menu, let's open it. I'll carefully click in the middle of the text 'Applications'/
+Action:
 ```python
 click(51, 8) 
 ```<end_code>
 
 Step 2:
 Short term goal: I want to open a text editor.
-Where I am: I am on the homepage of my desktop, with the applications menu open.
-What I see: I see an Accessories section, I see it is a section in the menu thanks to the tiny white triangle after the text accessories.
-Reflection: I think that a notes application would fit the Accessories section. I SHOULD NOT try to move through the menus with scroll, it won't work:.
-Action: I'll look for Accessories and click on it being very precise, clicking in the middle of the text 'Accessories'.
-Code:
+What I see: I am on the homepage of my desktop, with the applications menu open. I see an Accessories section, I see it is a section in the menu thanks to the tiny white triangle after the text accessories.
+Reflection: I think that a notes application would fit the Accessories section. I SHOULD NOT try to move through the menus with scroll, it won't work:
+I'll look for Accessories and click on it being very precise, clicking in the middle of the text 'Accessories'.
+Action:
 ```python
 click(76, 195) 
 ```<end_code>
 
 Step 3:
 Short term goal: I want to open a text editor.
-Where I am: I am under the Accessories menu.
-What I see: under the open submenu Accessories, I've found 'Text Editor'.
-Reflection: This must be my notes app. I remember that menus are navigated through clicking.
-Action: I will now click on it being very precise, clicking in the middle of the text 'Text Editor'.
-Code:
+What I see: I am under the Accessories menu. Under the open submenu Accessories, I've found 'Text Editor'.
+Reflection: This must be my notes app. I remember that menus are navigated through clicking. I will now click on it being very precise, clicking in the middle of the text 'Text Editor'.
+Action:
 ```python
 click(251, 441) 
 ```<end_code>
 
 Step 4:
 Short term goal: I want to open a text editor.
-Where I am: I am still under the Accessories menu.
-What I see: Nothing has changed compared to previous screenshot. Under the open submenu Accessories, I still see 'Text Editor'. The green cross is off from the element.
-Reflection: My last click must have been off. Let's correct this.
-Action: I will click the correct place, right in the middle of the element.
-Code:
+What I see: I am still under the Accessories menu. Nothing has changed compared to previous screenshot. Under the open submenu Accessories, I still see 'Text Editor'. The green cross is off from the element.
+Reflection: My last click must have been off. Let's correct this. I will click the correct place, right in the middle of the element.
+Action:
 ```python
 click(241, 441) 
 ```<end_code>
 
 Step 5:
 Short term goal: I want to type 'Hello World'.
-Where I am: I have opened a Notepad.
-What I see: The Notepad app is open on an empty page
+What I see: I have opened a Notepad. The Notepad app is open on an empty page
 Reflection: Now Notepad is open as intended, time to type text.
-Action: I will type the requested text.
-Code:
+Action:
 ```python
 type_text("Hello World")
 ```<end_code>
 
 Step 6:
 Short term goal: I want to type 'Hello World'.
-Where I am: I have opened a Notepad.
 What I see: The Notepad app displays 'Hello World'
-Reflection: Now that I've 1. Opened the notepad and 2. typed 'Hello World', and 3. the result seems correct, I think the Task is completed.
-Action: I will return a confirmation that the task is completed.
-Code:
+Reflection: Now that I've 1. Opened the notepad and 2. typed 'Hello World', and 3. the result seems correct, I think the Task is completed. I will return a confirmation that the task is completed.
+Action:
 ```python
 final_answer("Done")
 ```<end_code>
 </task_resolution_example>
 
-<click_guidelines>
-Look at elements on the screen to determine what to click or interact with.
-Use precise coordinates based on the current screenshot for mouse movements and clicks. 
-Whenever you click, MAKE SURE to click in the middle of the button, text, link or any other clickable element. Not under, not on the side. IN THE MIDDLE, else you risk to miss it.
-In menus it is always better to click in the middle of the text rather than in the tiny icon. Calculate extremelly well the coordinates. A mistake here can make the full task fail.
-The desktop has a resolution of <<resolution_x>>x<<resolution_y>> pixels: NEVER USE HYPOTHETIC OR ASSUMED COORDINATES, USE TRUE COORDINATES that you can see from the screenshot.
-Sometimes you may have missed a click, so never assume that you're on the right page, always make sure that your previous action worked. In the screenshot you can see if the mouse is out of the clickable area. Pay special attention to this.
-</click_guidelines>
-
 <general_guidelines>
 Always analyze the latest screenshot carefully before performing actions.
 You can wait for appropriate loading times using the wait() tool. But don't wait forever, sometimes you've just misclicked and the process didn't launch.
@@ -144,7 +120,7 @@ Use click to move through menus on the desktop and scroll for web and specific a
 Always analyze the latest screenshot carefully before performing actions.
 Desktop menus usually expand with more options, the tiny triangle next to some text in a menu means that menu expands. For example in Office in the Applications menu expands showing presentation or writing applications. 
 NEVER CLICK THE WEB BROWSER ICON TO OPEN THE WEB BROWSER: use open_url directly.
-In browser, ignore any sign in popups while they don't interfere with your usage of the browser.
+In browser, ignore any sign-in popups while they don't interfere with the elements you want to interact with.
 </general_guidelines>
 """.replace("<<current_date>>", datetime.now().strftime("%A, %d-%B-%Y"))
 
@@ -228,91 +204,6 @@ class E2BVisionAgent(CodeAgent):
         self._setup_desktop_tools()
         self.step_callbacks.append(self.take_screenshot_callback)
 
-    def initialize_system_prompt(self) -> str:
-        if False:
-            return """You are a desktop automation assistant that can control a remote desktop environment.
-You only have access to the following tools to interact with the desktop, no additional ones:
-- click(x, y): Performs a left-click at the specified coordinates
-- right_click(x, y): Performs a right-click at the specified coordinates
-- double_click(x, y): Performs a double-click at the specified coordinates
-- move_mouse(x, y): Moves the mouse cursor to the specified coordinates
-- type_text(text): Types the specified text at the current cursor position
-- press_key(key): Presses a keyboard key (e.g., "Return", "tab", "ctrl+c")
-- scroll(x, y, direction, amount): Scrolls a website in a browser or a document (direction can be "up" or "down", a common amount is 1 or 2 scroll("down",1) ). DO NOT use scroll to move through linux desktop menus. x, y, is the mouse position to scroll on.
-- wait(seconds): Waits for the specified number of seconds. Very useful in case the prior order is still executing (for example starting very heavy applications like browsers or office apps)
-- open_url(url): Directly opens a browser with the specified url, saves time compared to clicking in a browser and going through the initial setup wizard.
-- drag_and_drop(x1, y1, x2, y2): Clicks [x1, y1], drags mouse to [x2, y2], then releases click.
-- find_on_page_ctrl_f(search_string): Scroll the viewport to the first occurrence of the search string. This is equivalent to Ctrl+F.
-- final_answer("YOUR FINAL ANSWER TEXT"): Announces that the task requested is completed and provides a final text
-The desktop has a resolution of {resolution_x}x{resolution_y}.
-IMPORTANT:
-- Remember the tools that you have as those can save you time, for example open_url to enter a website rather than searching for the browser in the OS.
-- Whenever you click, MAKE SURE to click in the middle of the button, text, link or any other clickable element. Not under, not on the side. IN THE MIDDLE. In menus it is always better to click in the middle of the text rather than in the tiny icon. Calculate extremelly well the coordinates. A mistake here can make the full task fail.
-- To navigate the desktop you should open menus and click. Menus usually expand with more options, the tiny triangle next to some text in a menu means that menu expands. For example in Office in the Applications menu expands showing presentation or writing applications. 
-- Always analyze the latest screenshot carefully before performing actions. If you clicked somewhere in the previous action and in the screenshot nothing happened, make sure the mouse is where it should be. Otherwise you can see that the coordinates were wrong.
-You must proceed step by step:
-1. Understand the task thoroughly
-2. Break down the task into logical steps
-3. For each step:
-a. Analyze the current screenshot to identify UI elements
-b. Plan the appropriate action with precise coordinates
-c. Execute ONE action at a time using the proper tool
-d. Wait for the action to complete before proceeding
-After each action, you'll receive an updated screenshot. Review it carefully before your next action.
-COMMAND FORMAT:
-Always format your actions as Python code blocks. For example:
-```python
-click(250, 304)
-```<end_code>
-TASK EXAMPLE:
-For a task like "Open a text editor and type 'Hello World'":
-1- First, analyze the screenshot to find the Applications menu and click on it being very precise, clicking in the middle of the text 'Applications':
-```python
-click(52, 10) 
-```<end_code>
-2- Remembering that menus are navigated through clicking, after analyzing the screenshot with the applications menu open we see that a notes application probably fits in the Accessories section (we see it is a section in the menu thanks to the tiny white triangle after the text accessories). We look for Accessories and click on it being very precise, clicking in the middle of the text 'Accessories'. DO NOT try to move through the menus with scroll, it won't work:
-```python
-click(76, 195) 
-```<end_code>
-3- Remembering that menus are navigated through clicking, after analyzing the screenshot with the submenu Accessories open, look for 'Text Editor' and click on it being very precise, clicking in the middle of the text 'Text Editor':
-```python
-click(241, 441) 
-```<end_code>
-4- Once Notepad is open, type the requested text:
-```python
-type_text("Hello World")
-```<end_code>
-5- Task is completed:
-```python
-final_answer("Done")
-```<end_code>
-Remember to:
-Always wait for appropriate loading times
-Use precise coordinates based on the current screenshot
-Execute one action at a time
-Verify the result before proceeding to the next step
-Use click to move through menus on the desktop and scroll for web and specific applications.
-REMEMBER TO ALWAYS CLICK IN THE MIDDLE OF THE TEXT, NOT ON THE SIDE, NOT UNDER.
-""".format(resolution_x=self.width, resolution_y=self.height)
-        else:
-            print("USING v2 prompt")
-            system_prompt = populate_template(
-                self.prompt_templates["system_prompt"],
-                variables={
-                    "tools": self.tools,
-                    "managed_agents": self.managed_agents,
-                    "authorized_imports": (
-                        "You can import from any package you want."
-                        if "*" in self.authorized_imports
-                        else str(self.authorized_imports)
-                    ),
-                },
-            )
-            assert system_prompt != self.prompt_templates["system_prompt"], (
-                "Populating prompt template failed"
-            )
-            return system_prompt
-
     def _setup_desktop_tools(self):
         """Register all desktop tools"""
 
@@ -508,7 +399,7 @@ REMEMBER TO ALWAYS CLICK IN THE MIDDLE OF THE TEXT, NOT ON THE SIDE, NOT UNDER.
 
         current_step = memory_step.step_number
 
-        time.sleep(3.0)  # Let things happen on the desktop
+        time.sleep(2.5)  # Let things happen on the desktop
         screenshot_bytes = self.desktop.screenshot(format="bytes")
         image = Image.open(BytesIO(screenshot_bytes))
 
@@ -518,9 +409,9 @@ REMEMBER TO ALWAYS CLICK IN THE MIDDLE OF THE TEXT, NOT ON THE SIDE, NOT UNDER.
 
         image_copy = image.copy()
 
-        if getattr(self, "click_coordinates", None):
-            print("DRAWING MARKER")
-            image_copy = draw_marker_on_image(image_copy, self.click_coordinates)
+        # if getattr(self, "click_coordinates", None):
+        #     print("DRAWING MARKER")
+        #     image_copy = draw_marker_on_image(image_copy, self.click_coordinates)
 
         self.last_marked_screenshot = AgentImage(screenshot_path)
         print(f"Saved screenshot for step {current_step} to {screenshot_path}")

eval.py

@@ -74,7 +74,7 @@ def create_agent(data_dir, desktop, max_steps: int):
         desktop=desktop,
         max_steps=max_steps,
         verbosity_level=2,
-        planning_interval=10,
+        # planning_interval=10,
     )
 
 
@@ -346,7 +346,7 @@ def main():
         "gmaps": "Use Google Maps to find the Hugging Face HQ in Paris",
         "wiki": "Go to Wikipedia and find what happend on April 4th",
         "hello": "Write 'Hello World' in a text editor",
-        "commute": "Find out how long it takes to travel by train from Bern and Basel",
+        "commute": "Find out the travel time by train from Bern to Basel on Google Maps",
         "hf_space": "Go to Hugging Face Spaces and then find the Space flux.1 schnell. Use the space to generate an image of a GPU",
     }