DeepEval: автоматическая оценка LLM-вызовов через OpenAI SDK

Title:

URL Source: https://raw.githubusercontent.com/confident-ai/deepeval/main/examples/notebooks/openai.ipynb

Markdown Content: { "cells": [ { "cell_type": "markdown", "id": "4f286a60", "metadata": {}, "source": [ "## Using DeepEval with OpenAI\n", "\n", "This guide will help you to evalaute LLM calls using OpenAI SDK, both as a standalone LLM call and as a part of LLM application. DeepEval's OpenAI integrations takes care of generating LLM spans for OpenAI SDK calls and it is fully compatible with the native `observe` decorator. " ] }, { "cell_type": "code", "execution_count": null, "id": "124c28e0", "metadata": {}, "outputs": [], "source": [ "!pip install openai -U deepeval ipywidgets" ] }, { "cell_type": "code", "execution_count": null, "id": "4aa94394", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"<your-openai-api-key>\"" ] }, { "cell_type": "code", "execution_count": null, "id": "fbcb3b26", "metadata": {}, "outputs": [], "source": [ "from deepeval.openai import OpenAI\n", "\n", "client = OpenAI()" ] }, { "cell_type": "markdown", "id": "6f236ac7", "metadata": {}, "source": [ "### Evaluating OpenAI SDK as a standalone LLM call\n", "\n", "There are 3 simple steps to evaluate OpenAI SDK as a standalone LLM call:" ] }, { "cell_type": "markdown", "id": "8c42d53e", "metadata": {}, "source": [ "#### Create an evalaution dataset with goldens." ] }, { "cell_type": "code", "execution_count": null, "id": "87638cfa", "metadata": {}, "outputs": [], "source": [ "from deepeval.dataset import Golden, EvaluationDataset\n", "\n", "goldens = [\n", " Golden(\n", " input=\"What are the top 5 most popular palces to eat in New York City?\"\n", " ),\n", " Golden(input=\"What is the weather in Paris, France?\"),\n", "]\n", "\n", "dataset = EvaluationDataset(goldens=goldens)" ] }, { "cell_type": "markdown", "id": "fd14318e", "metadata": {}, "source": [ "#### Select the metrics to evaluate.\n", "\n", "Note: The current integrations only supports metrics with input, output and tools called. This means that the only eligible metrics are those which have required arguments as `input`, `output` and `tools_called`. However you can still set the other test cases parameters like (`expected_output` or `context`) in the next step." ] }, { "cell_type": "code", "execution_count": null, "id": "31540d2f", "metadata": {}, "outputs": [], "source": [ "from deepeval.metrics import AnswerRelevancyMetric, BiasMetric\n", "\n", "metrics = [AnswerRelevancyMetric(), BiasMetric()]" ] }, { "cell_type": "markdown", "id": "99e14530", "metadata": {}, "source": [ "### Run the evals \n", "\n", "The `evals_iterator` from `EvaluationDataset` object returns a generator of goldens. You can iterate through the goldens and run the evals. If you want to set more parameters for the test cases, you can set them in the `LlmSpanContext` object." ] }, { "cell_type": "code", "execution_count": null, "id": "85d27a82", "metadata": {}, "outputs": [], "source": [ "from deepeval.tracing import trace, LlmSpanContext\n", "\n", "for golden in dataset.evals_iterator():\n", " # run OpenAI client\n", " with trace(\n", " llm_span_context=LlmSpanContext(\n", " metrics=metrics,\n", " expected_output=golden.expected_output,\n", " )\n", " ):\n", " client.chat.completions.create(\n", " model=\"gpt-4o\",\n", " messages=[\n", " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", " {\"role\": \"user\", \"content\": golden.input},\n", " ],\n", " )" ] }, { "cell_type": "markdown", "id": "69584c14", "metadata": {}, "source": [ "### Evaluating OpenAI as SDK as a part of LLM application" ] }, { "cell_type": "code", "execution_count": null, "id": "5b2194c1", "metadata": {}, "outputs": [], "source": [ "from deepeval.tracing import observe\n", "\n", "\n", "@observe()\n", "def retrieve_docs(query):\n", " return [\n", " \"Paris is the capital and most populous city of France.\",\n", " \"It has been a major European center of finance, diplomacy, commerce, and science.\",\n", " ]\n", "\n", "\n", "@observe()\n", "def llm_app(input):\n", " with trace(\n", " llm_span_context=LlmSpanContext(\n", " metrics=[AnswerRelevancyMetric(), BiasMetric()],\n", " ),\n", " ):\n", " response = client.chat.completions.create(\n", " model=\"gpt-4o\",\n", " messages=[\n", " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"\\n\".join(retrieve_docs(input))\n", " + \"\\n\\nQuestion: \"\n", " + input,\n", " },\n", " ],\n", " )\n", " return response.choices[0].message.content" ] }, { "cell_type": "code", "execution_count": null, "id": "5ec0043f", "metadata": {}, "outputs": [], "source": [ "# Create dataset\n", "dataset = EvaluationDataset(\n", " goldens=[\n", " Golden(\n", " input=\"What are the top 5 most popular palces to eat in New York City?\"\n", " ),\n", " Golden(input=\"What is the weather in Paris, France?\"),\n", " ]\n", ")\n", "\n", "# Iterate through goldens\n", "for golden in dataset.evals_iterator():\n", " # run your LLM application\n", " llm_app(input=golden.input)" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 5 }