# Your leaderboard name
TITLE = """<h1 align="center" id="space-title">LLM Leaderboard for CRM</h1>
<h3>Assess which LLMs are accurate enough or need fine-tuning, and weigh this versus tradeoffs of speed, costs, and trust and safety. This is based on human manual and automated evaluation with real operational CRM data per use case.</h3>
"""

# What does your leaderboard evaluate?
INTRODUCTION_TEXT = """

"""

# Which evaluations are you running? how can people reproduce what you have?
LLM_BENCHMARKS_TEXT = """
1) GPT-4T was used except for some accuracy use cases with atypically long input tokens.
2) Hyperparameters were optimized for a subset of models evaluated (platform models?) Were parameters optimized as well?
3) Latency reflects the mean latency over a single time range on a high-speed internet connection; response times for external APIs may vary over time and be impacted by internet speed, location, etc.
3) Latency reflects the time to receive the entire completion.
4) Some external APIs were direct to the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic)
5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects
6) All tests on open source models were performed on original models (correct?); custom fine-tuning may impact performance in trust / safety / toxicity / bias / etc.
7) For the tests on latency, the inputs were *approximately* 500 / 3000 tokens. A short prompt was added and different models tokenize differently.
8) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic apis).
9) Something about limitations of LLM judges (despite correlation with human annotators)
10) Task-specific model variants were not used from the external providers (command-r is sort of retrieval specific, but this was not one of the use cases)
11) Maybe something about the tasks being primarily summarization / generation
12) CRM T&S is done by perturbing words: 1) for gender bias, we perturb person names and pronouns to opposite gender. 2) for entity bias, we perturb company names to its competitors in the same sector
13) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time.
"""

CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
CITATION_BUTTON_TEXT = r"""
@misc{crm-llm-leaderboard,
  author = {Salesforce AI},
  title = {LLM Leaderboard for CRM},
  year = {2024},
  publisher = {Salesforce AI},
  howpublished = "\url{https://https://huggingface.co/spaces/Salesforce/crm_llm_leaderboard}"
}
"""