Оценка агентов через DeepEval: метрики и датасет

Title:

URL Source: https://raw.githubusercontent.com/confident-ai/deepeval/main/examples/notebooks/pydantic_ai.ipynb

Markdown Content: { "cells": [ { "cell_type": "markdown", "id": "fe2fca83", "metadata": {}, "source": [ "## Evaluate Pydantic AI weather agent\n", "This tutorial will show you how to evaluate Pydantic AI agents using DeepEval's dataset iterator.\n" ] }, { "cell_type": "markdown", "id": "36ae4769", "metadata": {}, "source": [ "### Install dependencies:" ] }, { "cell_type": "code", "execution_count": null, "id": "b9cd4e3f", "metadata": {}, "outputs": [], "source": [ "!pip install pydantic-ai -U deepeval --quiet" ] }, { "cell_type": "markdown", "id": "3e90569a", "metadata": {}, "source": [ "### Set your OpenAI API key:" ] }, { "cell_type": "code", "execution_count": null, "id": "ac517022", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"<your-openai-api-key>\"" ] }, { "cell_type": "markdown", "id": "dbb85503", "metadata": {}, "source": [ "### Hyperparameters\n", "\n", "Hyperparameters of an LLM are the parameters that are used to control the behavior of the LLM application. It can be model, temperature, max tokens, or even you static prompts (for eg, system prompt). One of the main aim of performing evlauation is to find the best set of hyperparameters for a given agent.\n", "\n", "For this application, we are using model as one of the hyperparameter.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "95070436", "metadata": {}, "outputs": [], "source": [ "hyperparameter_model = \"gpt-4o\"" ] }, { "cell_type": "markdown", "id": "ff5d91c8", "metadata": {}, "source": [ "### Create a Pydantic AI agent. \n", "\n", "This is the same example as the one in the [Pydantic AI docs](https://ai.pydantic.dev/examples/weather-agent/). User can ask for the weather in multiple cities, the agent will use the `get_lat_lng` tool to get the latitude and longitude of the locations, then use\n", "the `get_weather` tool to get the weather." ] }, { "cell_type": "code", "execution_count": null, "id": "ffc9177b", "metadata": {}, "outputs": [], "source": [ "from __future__ import annotations as _annotations\n", "\n", "import asyncio\n", "from dataclasses import dataclass\n", "from typing import Any\n", "\n", "from httpx import AsyncClient\n", "from pydantic import BaseModel\n", "\n", "from pydantic_ai import Agent, RunContext\n", "\n", "\n", "@dataclass\n", "class Deps:\n", " client: AsyncClient\n", "\n", "\n", "weather_agent = Agent(\n", " hyperparameter_model,\n", " instructions=\"Be concise, reply with one sentence.\",\n", " deps_type=Deps,\n", " retries=2,\n", ")\n", "\n", "\n", "class LatLng(BaseModel):\n", " lat: float\n", " lng: float\n", "\n", "\n", "@weather_agent.tool\n", "async def get_lat_lng(\n", " ctx: RunContext[Deps], location_description: str\n", ") -> LatLng:\n", " \"\"\"Get the latitude and longitude of a location.\n", "\n", " Args:\n", " ctx: The context.\n", " location_description: A description of a location.\n", " \"\"\"\n", " # NOTE: the response here will be random, and is not related to the location description.\n", " r = await ctx.deps.client.get(\n", " \"https://demo-endpoints.pydantic.workers.dev/latlng\",\n", " params={\"location\": location_description},\n", " )\n", " r.raise_for_status()\n", " return LatLng.model_validate_json(r.content)\n", "\n", "\n", "@weather_agent.tool\n", "async def get_weather(\n", " ctx: RunContext[Deps], lat: float, lng: float\n", ") -> dict[str, Any]:\n", " \"\"\"Get the weather at a location.\n", "\n", " Args:\n", " ctx: The context.\n", " lat: Latitude of the location.\n", " lng: Longitude of the location.\n", " \"\"\"\n", " # NOTE: the responses here will be random, and are not related to the lat and lng.\n", " temp_response, descr_response = await asyncio.gather(\n", " ctx.deps.client.get(\n", " \"https://demo-endpoints.pydantic.workers.dev/number\",\n", " params={\"min\": 10, \"max\": 30},\n", " ),\n", " ctx.deps.client.get(\n", " \"https://demo-endpoints.pydantic.workers.dev/weather\",\n", " params={\"lat\": lat, \"lng\": lng},\n", " ),\n", " )\n", " temp_response.raise_for_status()\n", " descr_response.raise_for_status()\n", " return {\n", " \"temperature\": f\"{temp_response.text} °C\",\n", " \"description\": descr_response.text,\n", " }\n", "\n", "\n", "async def run_agent(input_query: str):\n", " async with AsyncClient() as client:\n", " deps = Deps(client=client)\n", " result = await weather_agent.run(input_query, deps=deps)\n", " return result.output\n", "\n", "\n", "await run_agent(\n", " \"What is the weather like in London and in Wiltshire?\"\n", ") # test run the agent" ] }, { "cell_type": "markdown", "id": "157564e3", "metadata": {}, "source": [ "### Evaluate the agent\n", "\n", "To evaluate Pydantic AI agents, use Deepeval's Pydantic AI `Agent` to supply metrics.\n", "\n", "\n", "> (Pro Tip) View your Agent's trace and publish test runs on [Confident AI](https://www.confident-ai.com/). Apart from this you get an in-house dataset editor and more advaced tools to monitor and enventually improve your Agent's performance. Get your API key from [here](https://app.confident-ai.com/)\n", "\n", "Given below is the code to instrument the application.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b4bc68ea", "metadata": {}, "outputs": [], "source": [ "# optional\n", "from deepeval.integrations.pydantic_ai import instrument_pydantic_ai\n", "\n", "instrument_pydantic_ai(api_key=\"<your-confident-api-key>\")" ] }, { "cell_type": "markdown", "id": "fa4b5044", "metadata": {}, "source": [ "### Dataset\n", "\n", "For evaluating the agent, we need a dataset. You can create your own dataset or use the one from the [Confident AI](https://www.confident-ai.com/docs/llm-evaluation/dataset-management/create-goldens).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e6476a9a", "metadata": {}, "outputs": [], "source": [ "from deepeval.dataset import EvaluationDataset\n", "\n", "dataset = EvaluationDataset()\n", "dataset.pull(alias=\"weather_agent_queries\", public=True)" ] }, { "cell_type": "markdown", "id": "c5b28c1f", "metadata": {}, "source": [ "### Create a metric to evaluate the agent.\n", "\n", "Deepeval provides a state of the art ready to use [metric](https://deepeval.com/docs/metrics-introduction) to evaluate the agent. For this example, we will use the `AnswerRelevancyMetric`.\n", "\n", "> [!NOTE]\n", "You can only run end-to-end evals on metrics that evaluate the input and actual output of your Pydantic agent.\n", "\n" ] }, { "cell_type": "markdown", "id": "ae8b395c", "metadata": {}, "source": [ "Using Deepeval's Pydantic AI `Agent` wrapper, you can supply metrics to the agent." ] }, { "cell_type": "code", "execution_count": null, "id": "d7d4f7cd", "metadata": {}, "outputs": [], "source": [ "from deepeval.integrations.pydantic_ai import Agent\n", "from deepeval.metrics import BaseMetric\n", "\n", "weather_agent = Agent(\n", " hyperparameter_model,\n", " instructions=\"Be concise, reply with one sentence.\",\n", " deps_type=Deps,\n", " retries=2,\n", ")\n", "\n", "\n", "class LatLng(BaseModel):\n", " lat: float\n", " lng: float\n", "\n", "\n", "@weather_agent.tool\n", "async def get_lat_lng(\n", " ctx: RunContext[Deps], location_description: str\n", ") -> LatLng:\n", " r = await ctx.deps.client.get(\n", " \"https://demo-endpoints.pydantic.workers.dev/latlng\",\n", " params={\"location\": location_description},\n", " )\n", " r.raise_for_status()\n", " return LatLng.model_validate_json(r.content)\n", "\n", "\n", "@weather_agent.tool\n", "async def get_weather(\n", " ctx: RunContext[Deps], lat: float, lng: float\n", ") -> dict[str, Any]:\n", "\n", " temp_response, descr_response = await asyncio.gather(\n", " ctx.deps.client.get(\n", " \"https://demo-endpoints.pydantic.workers.dev/number\",\n", " params={\"min\": 10, \"max\": 30},\n", " ),\n", " ctx.deps.client.get(\n", " \"https://demo-endpoints.pydantic.workers.dev/weather\",\n", " params={\"lat\": lat, \"lng\": lng},\n", " ),\n", " )\n", " temp_response.raise_for_status()\n", " descr_response.raise_for_status()\n", " return {\n", " \"temperature\": f\"{temp_response.text} °C\",\n", " \"description\": descr_response.text,\n", " }\n", "\n", "\n", "async def run_agent(input_query: str, metrics: list[BaseMetric]):\n", " async with AsyncClient() as client:\n", " deps = Deps(client=client)\n", " result = await weather_agent.run(\n", " input_query, deps=deps, metrics=metrics\n", " )\n", " return result.output" ] }, { "cell_type": "markdown", "id": "5fc5c34f", "metadata": {}, "source": [ "### Use the dataset iterator to evaluate the agent.\n", "\n", "Use the dataset iterator (from the dataset that was pulled earlier from the Confident AI) to evaluate the agent." ] }, { "cell_type": "code", "execution_count": null, "id": "6631c095", "metadata": {}, "outputs": [], "source": [ "from deepeval.metrics import AnswerRelevancyMetric\n", "\n", "for golden in dataset.evals_iterator():\n", " task = asyncio.create_task(\n", " run_agent(\n", " golden.input,\n", " metrics=[\n", " AnswerRelevancyMetric(\n", " threshold=0.7, model=\"gpt-4o\", include_reason=True\n", " )\n", " ],\n", " )\n", " )\n", " dataset.evaluate(task)" ] }, { "cell_type": "markdown", "id": "65086d9a", "metadata": {}, "source": [ "Try changing hyperparameters and see how the agent performs." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 5 }