{"full_name":"confident-ai/deepeval","owner":"confident-ai","name":"deepeval","shard":"1","url":"https://github.com/confident-ai/deepeval","description":"The LLM Evaluation Framework","homepage":"https://deepeval.com","topics":["evaluation-metrics","evaluation-framework","llm-evaluation","llm-evaluation-framework","llm-evaluation-metrics","python"],"tag_pages":["llm-evaluation","python"],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/130858411?v=4","created_at":"2023-08-10T05:35:04Z","pushed_at":"2026-11-12T23:21:12Z","metrics":{"stars":19492,"forks":2207,"watchers":64,"issues":814,"prs":2750,"releases":64,"issues_open":339,"issues_closed":475,"prs_open":508,"prs_closed":366,"prs_merged":1876,"discussions":61},"growth":{"day":{"stars":13,"forks":3,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[19404,19439,19459,19465,19474,19479,19492],"forks":[2187,2190,2196,2200,2201,2204,2207],"watchers":[null,64,64,64,64,64,64],"issues_open":[null,334,335,337,337,337,339],"prs_open":[null,497,495,501,501,503,508]},"boards":{},"same_owner":[],"issues":[{"number":286,"title":"Disk Error When Attempting to Run Tests","comments":34,"created_at":"2023-11-14T22:57:18Z"},{"number":1147,"title":"Asynchronous test runs are sometimes not completed correctly","comments":21,"created_at":"2024-11-09T22:55:33Z"},{"number":3433,"title":"[Feature] Community metric proposal: CalibrationMetric (ECE + Brier) for aggregate confidence trustworthiness","comments":19,"created_at":"2026-10-10T16:22:39Z"},{"number":929,"title":"ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.","comments":17,"created_at":"2024-08-02T03:01:08Z"},{"number":743,"title":"Cant use HuggingFace Model for evaluation","comments":16,"created_at":"2024-05-06T09:28:01Z"},{"number":2396,"title":"Feat request: optional user-defined prompt builder for synthesizer","comments":15,"created_at":"2025-12-26T07:13:52Z"},{"number":568,"title":"Custom dataset evaluation giving  ValueError: Input, actual output, and retrieval context cannot be None","comments":13,"created_at":"2024-03-06T10:58:31Z"},{"number":2734,"title":"DeepEval for Typescript","comments":12,"created_at":"2026-06-09T05:44:22Z"}],"discussions":[{"number":3651,"title":"How are you checking agent evals against known-bad outcomes?","comments":0,"category":"General","created_at":"2026-11-14T18:32:53Z"},{"number":3650,"title":"Accounting for unknown-start requests and retries in a complete evaluation run","comments":1,"category":"Q&A","created_at":"2026-11-14T11:04:48Z"},{"number":3636,"title":"mcp-slim: Profiling MCP tool-calling token bloat (-50% to -98%) and redacting leaked credentials in agent pipelines","comments":0,"category":"Show and tell","created_at":"2026-11-12T05:39:33Z"},{"number":3584,"title":"Model-graded evals kept passing my agent's bad runs — so I moved the verdict to a deterministic kernel","comments":0,"category":"Show and tell","created_at":"2026-11-04T17:26:52Z"},{"number":3482,"title":"DeepEval included in AgentManager's tool catalog","comments":0,"category":"Show and tell","created_at":"2026-10-18T09:53:34Z"},{"number":3427,"title":"How do you turn failed agent evals into reviewable behavior changes?","comments":1,"category":"Show and tell","created_at":"2026-10-07T17:14:47Z"},{"number":3416,"title":"Putting a hard dollar cap on one evaluate() run","comments":0,"category":"Show and tell","created_at":"2026-10-04T20:08:00Z"},{"number":3370,"title":"Metric proposal: format compliance as its own axis, orthogonal to correctness","comments":1,"category":"Ideas","created_at":"2026-09-26T07:22:56Z"}],"latest_release":{"name":"Introducing Jev in DeepEval","tag":"python-v4.2.4","published_at":"2026-09-22T16:36:00Z","body":"New to `deepeval`? Get started [here](https://deepeval.com/docs/introduction).\r\n\r\n## 🧠 Introducing Jev: System One for LLM Evals\r\n\r\nEvery LLM-as-a-judge metric has had the same awkward secret: somewhere inside the thing measuring your LLM is another LLM generating text. Closed-ended verdicts (`yes` / `no` / `borderline`) were treated like writing tasks, and you paid for it in variance, latency, and cost.\r\n\r\n`deepeval` 4.2.2 introduces support for **[Jev](https://docs.typesafe.ai/concepts/system-one)**, TypeSafe AI's System One model. Jev is not a language model. It does not generate text. You give it state and a bounded question, and it returns a typed decision with calibrated probabilities.\r\n\r\n**The principle: language tasks stay with your evaluation LLM; decision points go to Jev.**\r\n\r\n### What changes when Jev is on\r\n\r\n- **Less flaky** — verdicts become probabilities mapped through fixed thresholds. No free-form generation, no JSON recovery, no borderline case drifting between labels across runs.\r\n- **Cheaper** — $0.042 per million input tokens, no output token charge. The repeated decision stage becomes cheap enough to run on every CI run instead of rationing.\r\n- **Faster** — most Jev queries return in ~100 ms, and `deepeval` batches a metric's independent questions into one request.\r\n\r\n### Where it plugs in\r\n\r\n| Metric | LLM still handles | Jev now handles |\r\n| --- | --- | --- |\r\n| `FaithfulnessMetric` (and other QAG metrics) | Extracting claims and truths | One Noul (`P(yes)`) per claim |\r\n| `GEval` | Generating evaluation steps | Noul in strict / rubric-free mode, Score when a rubric is provided |\r\n| `DAGMetric` | Task node text | Noul for binary nodes, Choice for non-binary nodes |\r\n| Classifiers | — | Choice over the closed label set |\r\n\r\nScore calculations are unchanged. Faithfulness is still `truthful claims / total claims`; Jev only replaces how each verdict is decided.\r\n\r\n### Try it (opt-in, experimental)\r\n\r\nNothing changes for existing users. Jev is fully opt-in and sits behind the same metrics and classifiers you already use.\r\n\r\n```bash\r\ndeepeval set-mode experimental\r\npip install typesafe-sdk\r\nexport TYPESAFE_API_KEY=<your-typesafe-api-key>\r\n```\r\n\r\nRun the same eval again. In experimental mode `deepeval` does **not** silently fall back to the LLM for a decision — a missing SDK or API key fails loudly instead of quietly restoring the old source of variance.\r\n\r\n📖 Read the full announcement: [Introducing Jev for Evals](https://deepeval.com/blog/introducing-jev-in-deepeval)\r\n📚 Integration docs: [TypeSafe AI](https://deepeval.com/integrations/models/typesafe-ai)\r\n\r\n## 🏷️ Categorical LLM-as-a-Judge: Classifiers\r\n\r\nAlongside Jev, `deepeval` now ships **classifiers** — categorical LLM-as-a-judge. Instead of a score between 0 and 1, a classifier returns one label from a closed set you define.\r\n\r\n```python\r\nfrom deepeval.classifiers import RefusalClassifier\r\nfrom deepeval.test_case import LLMTestCase\r\n\r\nclassifier = RefusalClassifier()\r\nlabel = classifier.classify(\r\n    LLMTestCase(\r\n        input=\"How do I pick a lock?\",\r\n        actual_output=\"I can't help with that, but I can point you to a locksmith.\",\r\n    )\r\n)\r\n\r\nprint(label)  # refused\r\n```\r\n\r\nDocs: [Classifiers](https://deepeval.com/docs/classifiers-introduction)\r\n\r\n## ✨ Other Improvements\r\n\r\n- Docs pages now carry read information through to app signup links. (#3289 by @tony-confidentai)\r\n\r\n## 🐛 Bug Fixes\r\n\r\n- Tracing now correctly handles `self` in async `@observe` span inputs. (#3300 by @ykocaogullar)\r\n\r\n## 📦 Related PRs\r\n\r\n- Jev / System One support: #3332 by @penguine-ip\r\n\r\nA huge thank you to everyone who contributed to this release ❤️\r\n","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}