Create README.md

README.md

@@ -0,0 +1,191 @@
+---
+license: apache-2.0
+metrics:
+- accuracy
+base_model:
+- meta-llama/Llama-3.1-8B-Instruct
+datasets:
+- stanfordnlp/nnetnav-live
+---
+
+# Model Card for Llama8b-NNetNav-Live
+
+<!-- Provide a quick summary of what the model is/does. [Optional] -->
+LLama8b-NNetNav-WA is a [LLama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model that is instruct-tuned with [NNetNav-Live](https://huggingface.co/datasets/stanfordnlp/nnetnav-live) data collected via unsupervised exploration on 15 live websites, with a larger [LLama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) model.
+
+The live websites used for exploration are:
+- Allrecipes.com
+- amazon.com
+- apple.com
+- bbcnews.com
+- booking.com
+- dictionary.cambridge.org
+- coursera.org
+- espn.com
+- github.com
+- google.com/flights
+- google.com/maps
+- google.com
+- huggingface.co
+- wolframalpha.com
+
+More details about this model can be found in our paper: [NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild](https://arxiv.org/abs/2410.02907).
+
+
+##  Table of Contents
+
+- [Model Card for Llama8b-NNetNav-Live](#model-card-for--model_id-)
+- [Table of Contents](#table-of-contents)
+- [Model Details](#model-details)
+- [Results on Benchmarks](#results-on-benchmarks)
+- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+- [Training Details](#training-details)
+  - [Training Data](#training-data)
+  - [Training Procedure](#training-procedure)
+- [Environmental Impact](#environmental-impact)
+- [Technical Specifications [optional]](#technical-specifications-optional)
+  - [Model Architecture and Objective](#model-architecture-and-objective)
+  - [Compute Infrastructure](#compute-infrastructure)
+    - [Hardware](#hardware)
+    - [Software](#software)
+- [Citation](#citation)
+- [Model Card Authors](#model-card-authors-optional)
+- [Model Card Contact](#model-card-contact)
+- [How to Get Started with the Model](#how-to-get-started-with-the-model)
+
+## Model Details
+This model is intended to be used as a **web-agent** i.e. given an instruction such as "Upvote the post by user smurty123 on subreddit r/LocalLLaMA", and a web-url "reddit.com", the model can perform the task by executing a sequence of actions.
+
+The action space of the model is as follows:
+```plaintext
+Page Operation Actions:
+`click [id]`: This action clicks on an element with a specific id on the webpage.
+`type [id] [content] [press_enter_after=0|1]`: Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless press_enter_after is set to 0.
+`hover [id]`: Hover over an element with id.
+`press [key_comb]`:  Simulates the pressing of a key combination on the keyboard (e.g., Ctrl+v).
+`scroll [down|up]`: Scroll the page up or down.
+
+Tab Management Actions:
+`new_tab`: Open a new, empty browser tab.
+`tab_focus [tab_index]`: Switch the browser's focus to a specific tab using its index.
+`close_tab`: Close the currently active tab.
+
+URL Navigation Actions:
+`goto [url]`: Navigate to a specific URL.
+`go_back`: Navigate to the previously viewed page.
+`go_forward`: Navigate to the next page (if a previous 'go_back' action was performed).
+
+Completion Action:
+`stop [answer]`: Issue this action when you believe the task is complete. If the objective is to find a text-based answer, provide the answer in the bracket. If you believe the task is impossible to complete, provide the answer as "N/A" in the bracket.
+
+```
+
+## Results on Benchmarks
+
+This model gets the following results on WebArena and WebVoyager:
+
+| Model                  | WebArena (SR) | WebVoyager (SR) |
+|------------------------|--------------:|---------------:|
+| **GPT-4**             | **14.1**      | **33.5**      |
+| **llama8b-nnetnav-wa** | **9.5**      | **35.2**      |
+
+
+## Bias, Risks, and Limitations
+
+### **Bias**
+As with all ML models, **Llama8b-NNetNav-Live** inherits biases from its training data. Since the dataset is collected via unsupervised exploration on live websites, during a specific time-period (December 2024 to January 2025), it will reflect biases present in these websites at that time.
+
+- **Selection Bias:** The model is trained on interactions from a specific set of live websites, and may struggle on out-of-domain websites e.g. government websites.   
+- **Demographic Bias:** Our set of live-websites over-represent Western English-speaking users, and the model may perform worse on non-English or culturally distinct websites.  
+  - Example: A model trained mostly on U.S. e-commerce sites may navigate amazon.com effectively but may struggle with Flipkart (India) or Rakuten (Japan).
+
+If you are interested in training a NNetNav based agent for your own domain, please check out our [codebase](https://github.com/MurtyShikhar/NNetnav).
+
+### **Risks**
+#### **1. Unintended Actions**
+The model operates by executing web actions based on textual observation spaces, which may lead to unintended consequences when dealing with ambiguous or poorly structured websites.
+
+- If instructed to "delete all spam messages in my inbox," but the website has unusual button placement in the AXTree, the model might mistakenly delete important emails instead.
+- If asked to "buy the cheapest laptop on Amazon," the model might select an accessory instead of an actual laptop if the AXTree of the listing page has misleading layout
+
+#### **2. Security & Privacy Risks**
+Since the model interacts with external web content, there are significant risks related to unintentional data exposure, credential leaks, and interaction with harmful content.
+
+- If asked to "log into my Gmail and check unread emails," the model may type and submit credentials without realizing it, potentially exposing passwords.
+- A user asking the model to "search for free software downloads" might inadvertently lead to interactions with phishing or malware-hosting sites.
+
+#### **3. Adversarial Manipulation**
+Malicious websites can deceive the model by using **dark patterns**—UI/UX tricks that mislead users (or bots).
+
+- A fraudulent website may create **fake "Close" buttons** in the AXTree that actually trigger **downloads or pop-ups**. The model, thinking it's closing a window, may instead **click a malicious link**.
+- If asked to "unsubscribe from a newsletter," but the page uses **misleading button labels** in the AXTree (e.g., "Unsubscribe" actually means "Resubscribe"), the model could perform the opposite action.
+
+#### **4. Legal & Ethical Considerations**
+Web navigation often involves handling user-generated content, news, and e-commerce transactions, all of which pose ethical and legal challenges.
+
+- If instructed to "find the latest election results," the model might click on a misleading news source, potentially spreading misinformation.
+- If asked to "find the cheapest flight ticket," it could unintentionally violate terms of service by scraping restricted airline data.
+
+### **Limitations**
+#### **1. Generalization to Unseen Websites**
+This model is trained via interaction on 15 live-websites. These interactions were carried out between December 2024 to January 2025. If you intend to use the model on websites that are very distinct from these 15 sites, we highly suggest training your own NNetNav model. See our [codebase](https://github.com/MurtyShikhar/NNetnav) for more information.
+
+#### **2. Instruction Sensitivity**
+Vague instructions can lead to unintended actions.
+
+- "Find me the best laptop for gaming" is **subjective**, and the model might select a **random option** instead of following some criteria (e.g., GPU, refresh rate).
+
+#### **3. Performance on Long-Horizon Tasks**
+The model may struggle when tasks require **deep memory retention, complex multi-step planning, or backtracking**.
+
+- *Example:* When booking a hotel on a travel website, the model might navigate **through multiple filters and options** but forget previous selections when reaching the checkout page.
+
+#### **4. Token Limitations**
+The model's **maximum sequence length of 20k tokens** limits its ability to handle long, continuous web interactions.
+
+- *Example:* When filling a very long multi-step form, the model might forget earlier responses, leading to errors.
+
+
+## How to Get Started with the Model
+
+TODO
+
+## Training Details
+
+### Training Data
+
+This model was trained on the [NNetnav-Live](https://huggingface.co/datasets/stanfordnlp/nnetnav-live) dataset, which is comprised of ~5k synthetic demonstrations from 15 live websites
+
+### Training Procedure
+
+This model was trained for 2 epochs (roughly 4k gradient steps) with a batch size of 128, and a maximum sequence length of 20000.
+
+## Environmental Impact
+
+- **Hardware Type:** 4 H100 GPUs (80G)
+- **Hours used:** Roughly 2 days.
+- **Cloud Provider:** Stanford compute.
+- **Compute Region:** Stanford energy grid.
+
+
+## Technical Specifications
+
+This model was trained on a slurm cluster.
+
+### Hardware
+
+This model was trained on 4 H100s.
+
+### Software
+
+This model was fine-tuned with [Open-Instruct](https://github.com/allenai/open-instruct/tree/main)
+
+
+## Model Card Authors
+
+<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->
+Shikhar Murty
+
+## Model Card Contact
+
+[email protected]