prompts.py

acceptance_criteria = """# Acceptance Criteria

Acceptance of a submission to TMLR should be based on positive answers to the following two questions.

**Are the claims made in the submission supported by accurate, convincing and clear evidence?**

This is the most important criterion. This implies assessing the technical soundness as well as the clarity of the narrative and arguments presented.

Any gap between claims and evidence should be addressed by the authors. Often, this will lead reviewers to ask the authors to provide more evidence by running more experiments. However, this is not the only way to address such concerns. Another is simply for the authors to adjust (reduce) their claims.

**Would some individuals in TMLR's audience be interested in the findings of this paper?**

This is arguably the most subjective criterion, and therefore needs to be treated carefully. Generally, a reviewer that is unsure as to whether a submission satisfies this criterion should assume that it does.

Crucially, it should not be used as a reason to reject work that isn't considered “significant” or “impactful” because it isn't achieving a new state-of-the-art on some benchmark. Nor should it form the basis for rejecting work on a method considered not “novel enough”, as novelty of the studied method is not a necessary criteria for acceptance. We explicitly avoid these terms (“significant”, “impactful”, “novel”), and focus instead on the notion of “interest”. If the authors make it clear that there is something to be learned by some researchers in their area from their work, then the criterion of interest is considered satisfied. TMLR instead relies on certifications (such as “Featured” and “Outstanding”) to provide annotations on submissions that pertain to (more speculative) assertions on significance or potential for impact.

Here's an example on how to use the criteria above. A machine learning class report that re-runs the experiments of a published paper has educational value to the students involved. But if it doesn't surface generalizable insights, it is unlikely to be of interest to (even a subset of) the TMLR audience, and so could be rejected based on this criterion. On the other hand, a proper reproducibility report that systematically studies the robustness or generalizability of a published method and lays out actionable lessons for its audience could satisfy this criterion."""


review_format = """# Review Format

A review should have the following content.

**Summary of contributions** Brief description, in the reviewer's words, of the contributions and new knowledge presented by the submission.

**Strengths and weaknesses** List of the strong aspects of the submission as well as weaker elements (if any) that you think require attention from the authors.

**Requested changes** List of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view.

**Broader impact concerns** Brief description of any concerns on the ethical implications of the work that would require adding a Broader Impact Statement (if one is not present) or that are not sufficiently addressed in the Broader Impact Statement section (if one is present). This part should be very brief (one sentence maximum)."""


review_prompt = f"""Your job is to provide a high-quality review of a paper for TMLR. 
Ensure that your review is constructive, actionable, and aligns with the standards of TMLR. 

Use the following guidelines:
1. Here is the acceptance criteria of TMLR: ```{acceptance_criteria}```
2. Adhere to the following review format: ```{review_format}```

In each of the review points, make sure to be specific and detailed. Make specific references to Figures and Tables in the paper to support your points.

Provide the review in the review format, and make sure to follow the acceptance criteria.

First, within the tags <thoughts>...</thoughts>, think about what to write in the review, focusing on potential positive and negative points to make.
Reason through which ones are the most reasonable and most important to include.

Then, within the tags <output>...</output>, write the review in the review format.
"""

combine_prompt = """Your job is to act as a meta-reviewer for a TMLR paper review.
For the given PDF, you are given several reviews of the paper.

Your job is to combine the following reviews into one cohesive review that is better than the individual reviews.
Do not shorten the original review. Make sure each review point is a full sentence that explains relevant details.

Think about which points are the strongest and fairest, and how to best present them in the review with clarity.

The combined review should be a full, standalone review that does not reference the original reviews.

Group together the points from each subsection according to the review format: ```{review_format}```

Here are all the reviews:
```{all_reviews_string}```

First, within the tags <thoughts>...</thoughts>, think through which points are the strongest and fairest, and how to best present them in the review with clarity.

Then, within the tags <output>...</output>, write the review in the review format.
"""

defend_prompt = """
You are given the PDF of a paper and the review for that paper for TMLR.

Your job is to defend and ask for clarifications for the paper against the following reviewer's criticisms, as if you were the author of the paper.
Point out criticisms which are not particularly actionable or reasonable, or are based off an incorrect understanding of the paper.
Also, ask for more more details whenever the critiques are ambiguous or too generic.

Provide clear reasons for your defense, citing specific parts of the paper.

Here is the review:
```{combined_review}```

Give some guidance as to how the feedback could be more constructive, actionable, or relevant given what you know as the author.
Point out parts of the criticism which are fair and constructive - do not just dismiss criticisms out of hand or be combative.
"""

revise_prompt = """Improve a paper review by incorporating one round of feedback from the author.

Reason through each of the criticisms and corresponding responses of the original review point by point, and modify the review accordingly.
Remove criticisms where the author's defense points out legitimate flaws in the original criticism.
Respond to requests for clarifications, and provide more details to strengthen the review.
Use this defense to strengthen the review by being more specific and detailed that go beyond the original criticism.
For example, if the original criticism was "The authors do not provide enough experimental validation", the defense might be "The authors provide ablation studies and empirical evaluations on several datasets, showing that the method is effective and generalizable".

Use this time to provide necessary context for each criticism made. Provide at least one extra sentence of context for each criticism, motivating it properly.
If there is a requested change, try to give an example of what that might entail rather than just stating what the reviewer asked for.

Produce the revised review in the same format as the original review.
Here is the review format that was used: ```{review_format}```

Here is the original review:
```{combined_review}```

Here is the rebuttal of the review:
```{defended_paper}```

Before producing the modified review, think step by step within the tags <thinking>...</thinking>.
Then, produce the modified review in the tags, with all the sections filled out and necessary context: <output>...</output>.
"""


human_style = """
Here is a review of a paper for TMLR:
```{review}```

Help re-style this review to be more human-like while maintaining a professional and objective tone. 

Keep all the core content the same and only change the style.

You can use the following techniques to make the review more human-like:

1. Use of first person singular, like "I wonder" or "I think".
2. Use of variability in sentence structure, like short and long sentences.
3. Sprinkle in 3-5 typos or minor errors. It should not be gramatically or syntactically perfect.
4. Don't use too many bullet points when the same idea can be expressed in a single sentence.

Remember, this should still sound professional and expert-level.

Be objective and do not reflect any personal feelings about the paperin the review (eg. do not say "I was amazed by the results").

Finally, do not use the following words in the review:
commendable innovative meticulous intricate notable versatile noteworthy invaluable pivotal potent
fresh ingenious cogent ongoing tangible profound methodical laudable lucid appreciable
fascinating adaptable admirable refreshing proficient intriguing thoughtful credible exceptional digestible
prevalent interpretative remarkable seamless economical proactive interdisciplinary sustainable optimizable comprehensive
vital pragmatic comprehensible unique fuller authentic foundational distinctive pertinent valuable
invasive speedy inherent considerable holistic insightful operational substantial compelling technological
beneficial excellent keen cultural unauthorized strategic expansive prospective vivid consequential
manageable unprecedented inclusive asymmetrical cohesive replicable quicker defensive wider imaginative
traditional competent contentious widespread environmental instrumental substantive creative academic sizeable
extant demonstrable prudent practicable signatory continental unnoticed automotive minimalistic intelligent meticulously reportedly lucidly innovatively aptly
methodically excellently compellingly impressively undoubtedly scholarly strategically intriguingly competently intelligently
hitherto thoughtfully profoundly undeniably admirably creatively logically markedly thereby contextually
distinctly judiciously cleverly invariably successfully chiefly refreshingly constructively inadvertently effectively
intellectually rightly convincingly comprehensively seamlessly predominantly coherently evidently notably professionally
subtly synergistically productively purportedly remarkably
traditionally starkly promptly richly nonetheless elegantly smartly solidly inadequately effortlessly
forth firmly autonomously duly critically immensely beautifully maliciously finely succinctly
further robustly decidedly conclusively diversely exceptionally concurrently appreciably methodologically universally
thoroughly soundly particularly elaborately uniquely neatly definitively substantively usefully adversely primarily
principally discriminatively efficiently scientifically
alike herein additionally subsequently potentially
```
In your response, only provide the re-styled review. Do not include any other text.
"""

formatting_prompt = """Reformat this review. Do not change any of the actual content, but reformat bullet points, line breaks, etc. such that it is natural and easy to read.

Review: ```{review}```

Please keep the original Markdown tags, like bold (two asterisks) or linebreak. 
After parsing the importance, remove the original importance tag. 
Shorten the Impact Concerns to a single sentence only.

First, within the tags <thoughts>...</thoughts>, think through ways to make the review sound more human-like.
Then, within the tags <output>...</output>, write the review in the review format.
"""