Spaces:

DesertWolf
/

test3

Configuration error

File size: 13,149 Bytes

447ebeb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LLM Ops Stack  - LiteLLM Proxy + Langfuse \n",
    "\n",
    "This notebook demonstrates how to use LiteLLM Proxy with Langfuse \n",
    "- Use LiteLLM Proxy for calling 100+ LLMs in OpenAI format\n",
    "- Use Langfuse for viewing request / response traces \n",
    "\n",
    "\n",
    "In this notebook we will setup LiteLLM Proxy to make requests to OpenAI, Anthropic, Bedrock and automatically log traces to Langfuse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup LiteLLM Proxy\n",
    "\n",
    "### 1.1 Define .env variables \n",
    "Define .env variables on the container that litellm proxy is running on.\n",
    "```bash\n",
    "## LLM API Keys\n",
    "OPENAI_API_KEY=sk-proj-1234567890\n",
    "ANTHROPIC_API_KEY=sk-ant-api03-1234567890\n",
    "AWS_ACCESS_KEY_ID=1234567890\n",
    "AWS_SECRET_ACCESS_KEY=1234567890\n",
    "\n",
    "## Langfuse Logging \n",
    "LANGFUSE_PUBLIC_KEY=\"pk-lf-xxxx9\"\n",
    "LANGFUSE_SECRET_KEY=\"sk-lf-xxxx9\"\n",
    "LANGFUSE_HOST=\"https://us.cloud.langfuse.com\"\n",
    "```\n",
    "\n",
    "\n",
    "### 1.1 Setup LiteLLM Proxy Config yaml \n",
    "```yaml\n",
    "model_list:\n",
    "  - model_name: gpt-4o\n",
    "    litellm_params:\n",
    "      model: openai/gpt-4o\n",
    "      api_key: os.environ/OPENAI_API_KEY\n",
    "  - model_name: claude-3-5-sonnet-20241022\n",
    "    litellm_params:\n",
    "      model: anthropic/claude-3-5-sonnet-20241022\n",
    "      api_key: os.environ/ANTHROPIC_API_KEY\n",
    "  - model_name: us.amazon.nova-micro-v1:0\n",
    "    litellm_params:\n",
    "      model: bedrock/us.amazon.nova-micro-v1:0\n",
    "      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID\n",
    "      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY\n",
    "\n",
    "litellm_settings:\n",
    "  callbacks: [\"langfuse\"]\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Make LLM Requests to LiteLLM Proxy\n",
    "\n",
    "Now we will make our first LLM request to LiteLLM Proxy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Setup Client Side Variables to point to LiteLLM Proxy\n",
    "Set `LITELLM_PROXY_BASE_URL` to the base url of the LiteLLM Proxy and `LITELLM_VIRTUAL_KEY` to the virtual key you want to use for Authentication to LiteLLM Proxy. (Note: In this initial setup you can)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "LITELLM_PROXY_BASE_URL=\"http://0.0.0.0:4000\"\n",
    "LITELLM_VIRTUAL_KEY=\"sk-oXXRa1xxxxxxxxxxx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatCompletion(id='chatcmpl-B0sq6QkOKNMJ0dwP3x7OoMqk1jZcI', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Langfuse is a platform designed to monitor, observe, and troubleshoot AI and large language model (LLM) applications. It provides features that help developers gain insights into how their AI systems are performing, make debugging easier, and optimize the deployment of models. Langfuse allows for tracking of model interactions, collecting telemetry, and visualizing data, which is crucial for understanding the behavior of AI models in production environments. This kind of tool is particularly useful for developers working with language models who need to ensure reliability and efficiency in their applications.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739550502, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_523b9b6e5f', usage=CompletionUsage(completion_tokens=109, prompt_tokens=13, total_tokens=122, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import openai\n",
    "client = openai.OpenAI(\n",
    "    api_key=LITELLM_VIRTUAL_KEY,\n",
    "    base_url=LITELLM_PROXY_BASE_URL\n",
    ")\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"gpt-4o\",\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"what is Langfuse?\"\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 View Traces on Langfuse\n",
    "LiteLLM will send the request / response, model, tokens (input + output), cost to Langfuse.\n",
    "\n",
    "![image_description](litellm_proxy_langfuse.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Call Anthropic, Bedrock models \n",
    "\n",
    "Now we can call `us.amazon.nova-micro-v1:0` and `claude-3-5-sonnet-20241022` models defined on your config.yaml both in the OpenAI request / response format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatCompletion(id='chatcmpl-7756e509-e61f-4f5e-b5ae-b7a41013522a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"Langfuse is an observability tool designed specifically for machine learning models and applications built with natural language processing (NLP) and large language models (LLMs). It focuses on providing detailed insights into how these models perform in real-world scenarios. Here are some key features and purposes of Langfuse:\\n\\n1. **Real-time Monitoring**: Langfuse allows developers to monitor the performance of their NLP and LLM applications in real time. This includes tracking the inputs and outputs of the models, as well as any errors or issues that arise during operation.\\n\\n2. **Error Tracking**: It helps in identifying and tracking errors in the models' outputs. By analyzing incorrect or unexpected responses, developers can pinpoint where and why errors occur, facilitating more effective debugging and improvement.\\n\\n3. **Performance Metrics**: Langfuse provides various performance metrics, such as latency, throughput, and error rates. These metrics help developers understand how well their models are performing under different conditions and workloads.\\n\\n4. **Traceability**: It offers detailed traceability of requests and responses, allowing developers to follow the path of a request through the system and see how it is processed by the model at each step.\\n\\n5. **User Feedback Integration**: Langfuse can integrate user feedback to provide context for model outputs. This helps in understanding how real users are interacting with the model and how its outputs align with user expectations.\\n\\n6. **Customizable Dashboards**: Users can create custom dashboards to visualize the data collected by Langfuse. These dashboards can be tailored to highlight the most important metrics and insights for a specific application or team.\\n\\n7. **Alerting and Notifications**: It can set up alerts for specific conditions or errors, notifying developers when something goes wrong or when performance metrics fall outside of acceptable ranges.\\n\\nBy providing comprehensive observability for NLP and LLM applications, Langfuse helps developers to build more reliable, accurate, and user-friendly models and services.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739554005, model='us.amazon.nova-micro-v1:0', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=380, prompt_tokens=5, total_tokens=385, completion_tokens_details=None, prompt_tokens_details=None))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import openai\n",
    "client = openai.OpenAI(\n",
    "    api_key=LITELLM_VIRTUAL_KEY,\n",
    "    base_url=LITELLM_PROXY_BASE_URL\n",
    ")\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"us.amazon.nova-micro-v1:0\",\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"what is Langfuse?\"\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Advanced - Set Langfuse Trace ID, Tags, Metadata \n",
    "\n",
    "Here is an example of how you can set Langfuse specific params on your client side request. See full list of supported langfuse params [here](https://docs.litellm.ai/docs/observability/langfuse_integration)\n",
    "\n",
    "You can view the logged trace of this request [here](https://us.cloud.langfuse.com/project/clvlhdfat0007vwb74m9lvfvi/traces/567890?timestamp=2025-02-14T17%3A30%3A26.709Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatCompletion(id='chatcmpl-789babd5-c064-4939-9093-46e4cd2e208a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"Langfuse is an observability platform designed specifically for monitoring and improving the performance of natural language processing (NLP) models and applications. It provides developers with tools to track, analyze, and optimize how their language models interact with users and handle natural language inputs.\\n\\nHere are some key features and benefits of Langfuse:\\n\\n1. **Real-Time Monitoring**: Langfuse allows developers to monitor their NLP applications in real time. This includes tracking user interactions, model responses, and overall performance metrics.\\n\\n2. **Error Tracking**: It helps in identifying and tracking errors in the model's responses. This can include incorrect, irrelevant, or unsafe outputs.\\n\\n3. **User Feedback Integration**: Langfuse enables the collection of user feedback directly within the platform. This feedback can be used to identify areas for improvement in the model's performance.\\n\\n4. **Performance Metrics**: The platform provides detailed metrics and analytics on model performance, including latency, throughput, and accuracy.\\n\\n5. **Alerts and Notifications**: Developers can set up alerts to notify them of any significant issues or anomalies in model performance.\\n\\n6. **Debugging Tools**: Langfuse offers tools to help developers debug and refine their models by providing insights into how the model processes different types of inputs.\\n\\n7. **Integration with Development Workflows**: It integrates seamlessly with various development environments and CI/CD pipelines, making it easier to incorporate observability into the development process.\\n\\n8. **Customizable Dashboards**: Users can create custom dashboards to visualize the data in a way that best suits their needs.\\n\\nLangfuse aims to help developers build more reliable, accurate, and user-friendly NLP applications by providing them with the tools to observe and improve how their models perform in real-world scenarios.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739554281, model='us.amazon.nova-micro-v1:0', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=346, prompt_tokens=5, total_tokens=351, completion_tokens_details=None, prompt_tokens_details=None))"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import openai\n",
    "client = openai.OpenAI(\n",
    "    api_key=LITELLM_VIRTUAL_KEY,\n",
    "    base_url=LITELLM_PROXY_BASE_URL\n",
    ")\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"us.amazon.nova-micro-v1:0\",\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"what is Langfuse?\"\n",
    "        }\n",
    "    ],\n",
    "    extra_body={\n",
    "        \"metadata\": {\n",
    "            \"generation_id\": \"1234567890\",\n",
    "            \"trace_id\": \"567890\",\n",
    "            \"trace_user_id\": \"user_1234567890\",\n",
    "            \"tags\": [\"tag1\", \"tag2\"]\n",
    "        }\n",
    "    }\n",
    ")\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## "
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}