updated prompts based on Mert feedback

app.py

@@ -293,19 +293,17 @@ agent_prompt = """
 You are given a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. Your task is to provide constructive feedback to the reviewer so that it becomes a high-quality review. You will do this by evaluating the review against a checklist and providing specific feedback about where the review fails. 
 
 Here are step-by-step instructions:
-1. Read the review and the paper 
-    - Carefully read through the text of the review,
-    - Then, read the paper about which the review was written to understand its content and context.
+1. Read the text of the review and the paper about which the review was written.
 
 2. Evaluate every comment in the review:
    - Focus on comments made in the "weaknesses" or "questions" sections of the review. Ignore the "summary" and "strengths" sections.
-   - For each comment, evaluate it against the following checklist. Examples are provided of good responses.
+   - For each comment, evaluate it against the following checklist. Follow the examples for how to respond.
 
      Checklist:
-     1. Check if the reviewer requests something that is OBVIOUSLY already present in the paper. Only proceed if you are ABSOLUTELY certain the reviewer made an error, otherwise do NOT comment on their point. If you find such a case, point the reviewer specifically to the relevant part of the paper through direct quoting. Only send a quote if it verbatim exists in the paper or review. Ensure that the quote from the paper specifically answers the claim or request made by the reviewer, and explain why the selected quote is relevant.
+     1. Check if the reviewer requests something obviously present in the paper. Only respond if certain of the reviewer's error. If so, quote the relevant paper section verbatim using <quote> </quote> tags and explain how it addresses the reviewer's point. Use only exact quotes and don't comment if uncertain.
         - Example 1: 
-          - **Reviewer Comment:** *..In Figure 4, the efficiency experiments have no results for Transformer models, which is a key limitation of the paper.I*
-          - **Feedback to reviewer:** You may want to check Section 3, Figure 5 of the paper which has the Transformer results. See: “In Transformers, the proposed technique provides 25% relative improvement in wall-clock time (Figure 5)”.
+          - **Reviewer Comment:** *..In Figure 4, the efficiency experiments have no results for Transformer models, which is a key limitation of the paper.* 
+          - **Feedback to reviewer:** You may want to check Section 3, Figure 5 of the paper which has the Transformer results. See: <quote> In Transformers, the proposed technique provides 25% relative improvement in wall-clock time (Figure 5) </quote>.
         - Example 2:
             - **Reviewer Comment:** *Figure 2. Are people forced to select a choice or could they select 'I don't know'? Did you monitor response times to see if the manipulated images required longer times for individuals to pass decisions? In Appendix A, you mention “the response times will also be released upon publication”, however, I do not see further information about this in the paper.*
             - As the reviewer already refers to the key part of the paper with quotes and asks a question that is not answered in the paper, we do not need to give feedback to this comment.
@@ -344,17 +342,14 @@ Remember:
 - Do not summarize your feedback at the end or include a preamble at the beginning.
 - Do not repeat anything the reviewer already included in their review.
 - Do not mention that you are using a checklist or guidelines.
-- DO not address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer.
+- Do not address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer.
 """
 
 critic_prompt = f"""
 You are given a list of feedback about a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. The aim of the feedback is to guide a reviewer to make the review high-quality. Your task is to edit the feedback for correctness and clarity.
 
 Here are step-by-step instructions:
-1. Read the review, the feedback list, and the paper 
-    - Carefully read through the text of the review
-    - Then, read the feedback list provided for that review
-    - Finally, read the paper about which the review was written to understand its content and context.
+1. Read the text of the review, the feedback list provided for that review, and the paper about which the review was written.
 
 2. Evaluate every piece of feedback in the feedback list:
    - For each feedback item, it is imperative that you evaluate the correctness of the feedback. If there is a quote in the feedback, ensure that the quote appears VERBATIM in the paper. You need to check every quote and factual claim in the feedback and edit for correctness, as it is imperative all the feedback is correct.
@@ -368,12 +363,14 @@ Here are step-by-step instructions:
         - Feedback: {{your short feedback}}
     - If you do not identify any issues with a comment-feedback pair, do not edit it.
 
+4. Remove any comment-feedback pairs where the feedback is that there is no feedback or the comment is good. The feedback should only be about edits that need to be made.
+
 Remember:
 - Be concise, limiting your feedback for each comment to 1-2 sentences.
 - Do not summarize your feedback at the end or include a preamble at the beginning.
 - Do not repeat anything the reviewer already included in their review.
 - Do not mention that you are using a checklist or guidelines.
-- DO not address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer.
+- Do not address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer.
 
 Here are the guidelines that were followed to generate the feedback list originally; you should adhere to these guidelines: {agent_prompt}.
 """
@@ -382,10 +379,7 @@ aggregator_prompt = f"""
 You are given multiple lists of feedback about a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. The aim of the feedback is to guide a reviewer to make the review high-quality. Your task is to aggregate the lists of feedback into one list.
 
 Here are step-by-step instructions:
-1. Read the review, the multiple feedback lists, and the paper 
-    - Carefully read through the text of the review
-    - Then, read each of the feedback lists provided for that review
-    - Finally, read the paper about which the review was written to understand its content and context.
+1. Read the text of the review, the multiple feedback lists provided for that review, and the paper about which the review was written.
 
 2. For all feedback lists, aggregate them into one list with the best comment-feedback pairs from each list.
     - For each comment-feedback pair in the multiple lists that are similar, determine which provides the best feedback and keep only that pair.
@@ -457,6 +451,7 @@ if user_input:
         best_feedback = create_feedback(review, pdf_text, agent_prompt, model)
     
     revised_feedback = critic(review, best_feedback, pdf_text, critic_prompt, model)
+    revised_feedback = revised_feedback.replace("<quote>", "'").replace("</quote>", "'")
 
     st.title(f'Review feedback')