Is there a demo for desktop?
Hey, I noticed the infer demo under infer_SSAgent-8B.py is designed for mobile app, do you have any examples for the desktop apps(web browser...)
Thank you for your interest in our work.
To perform inference in a desktop scenario, you need to:
a) Update the task description;
b) Replace the image file with a desktop screenshot;
c) Update the definition of the action space;
d) If it's not the first step, you need to supplement the historical actions.
Currently, the demo is only for single-step inference. We will roll out an automated inference framework in the future.
Thank you for your interest in our work.
To perform inference in a desktop scenario, you need to:
a) Update the task description;
b) Replace the image file with a desktop screenshot;
c) Update the definition of the action space;
d) If it's not the first step, you need to supplement the historical actions.Currently, the demo is only for single-step inference. We will roll out an automated inference framework in the future.
Thx!
I would like to make sure: is this model robust to a custom action space? That is, no necessary to follow the names and parameters that were used during training? If not, could you please provide the recommend action space for desktop?
Sure. Here is an action space for the Mind2Web dataset for your reference:
- CLICK([block_index, cx, cy, w, h], "text")
- SELECT([block_index, cx, cy, w, h], "text")
- TYPE("text")