{"full_name":"vibrantlabsai/ragas","owner":"vibrantlabsai","name":"ragas","shard":"1","url":"https://github.com/vibrantlabsai/ragas","description":"Supercharge Your LLM Application Evaluations 🚀","homepage":"https://docs.ragas.io","topics":["llm","llmops","evaluation"],"tag_pages":["evaluation","llm","llmops"],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/122604797?v=4","created_at":"2023-05-08T17:48:04Z","pushed_at":"2026-11-14T03:06:20Z","metrics":{"stars":16328,"forks":1821,"watchers":61,"issues":1366,"prs":1729,"releases":91,"issues_open":284,"issues_closed":1082,"prs_open":268,"prs_closed":328,"prs_merged":1133,"discussions":0},"growth":{"day":{"stars":8,"forks":3,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[16280,16296,16305,16306,16313,16320,16328],"forks":[1806,1809,1813,1813,1814,1818,1821],"watchers":[null,61,61,61,61,61,61],"issues_open":[null,392,392,290,286,284,284],"prs_open":[null,270,271,271,266,267,268]},"boards":{},"same_owner":[],"issues":[{"number":1447,"title":"AttributeError","comments":29,"created_at":"2024-10-08T12:47:26Z"},{"number":1435,"title":"How to adapt language of result just exactly the same to the contexts given which is separated into several files and the files are in different languages.  ","comments":23,"created_at":"2024-10-06T08:50:18Z"},{"number":1322,"title":"How does RAGAS generate ground_truth in TestSetGenerator class","comments":23,"created_at":"2024-09-16T05:53:52Z"},{"number":1439,"title":"subclass of LLM","comments":20,"created_at":"2024-10-07T09:29:34Z"},{"number":1381,"title":"Can't generate testdataset, always connection error and event loops is closed.","comments":20,"created_at":"2024-09-30T03:48:36Z"},{"number":1121,"title":"Custom prompt for test set generation","comments":20,"created_at":"2024-07-22T14:55:20Z"},{"number":1807,"title":"expectation error","comments":18,"created_at":"2025-01-04T21:44:22Z"},{"number":1150,"title":"Getting error \"Failed to parse output. Returning None\" on faithfulness metric","comments":15,"created_at":"2024-08-01T08:15:19Z"}],"discussions":[],"latest_release":{"name":"v0.4.4","tag":"v0.4.4","published_at":"2026-11-14T03:06:20Z","body":"This release restores a working install, fixes a published security vulnerability, and makes Ragas work again with current OpenAI and Anthropic models.\n\n```bash\npip install -U ragas\n```\n\n## Highlights\n\n- **`import ragas` works again.** A fresh install pulled in `langchain-community` 0.4, which removed the VertexAI modules Ragas imported at start-up, so the import failed with `ModuleNotFoundError`. Those imports are now optional. (#2956)\n- **Security fix for multimodal metrics (CVE-2026-6587).** See the security section below. (#3105)\n- **Anthropic works on current Claude models.** Ragas no longer sends the sampling defaults that current models reject, and it no longer forces a tool call that the newest models refuse. (#3106, #3108)\n- **OpenAI models with dotted versions work.** `gpt-5.2` and similar are now recognised, so `max_tokens` is sent as `max_completion_tokens`. (#2700)\n- **OpenAI models on Amazon Bedrock work.** Bedrock model IDs such as `us.openai.gpt-6-sol` are now recognised as reasoning models, so they no longer fail with a 400. (#3049)\n- **Synchronous clients work in async metrics.** `agenerate()` with a sync client runs in a thread instead of raising `TypeError`. (#2657)\n- **Deprecation warnings point to imports that exist.** 22 of the warnings in `ragas.metrics` suggested a `ragas.metrics.collections` import that raised `ImportError`. (#2974)\n\n## Security\n\n`MultiModalFaithfulness` and `MultiModalRelevance` load images named in `retrieved_contexts`. A context string could make the machine running the evaluation fetch an internal URL (for example a cloud metadata endpoint) or read a local image file, and send the result to the LLM provider. This is CVE-2026-6587 / GHSA-95ww-475f-pr4f.\n\nBoth the collections API and the legacy API are fixed:\n\n- Image URLs are fetched only from public internet addresses. The check runs on the address actually connected to, on every redirect, so redirects and DNS rebinding do not get around it.\n- Local image paths are not read unless you enable local file access and name the one directory images may be loaded from.\n\nIf you use these metrics with untrusted datasets, upgrade.\n\nThanks to @GeoffreyManda, whose fix and tests in #3089 this builds on, and to the reporters in #2754.\n\n## Changes you may notice\n\n**Multimodal metrics**\n\n- Local image paths in `retrieved_contexts` are no longer loaded by default in the collections API. An image that is not loaded is passed to the model as text and a warning is logged, so check your logs if scores change.\n- Image URLs on `localhost` or private addresses are refused in both APIs.\n- To opt back in, set these before running the metric:\n\n  ```python\n  import ragas.prompt.multi_modal_prompt as image_policy\n\n  # Read local images, but only from inside this directory\n  image_policy.ALLOW_LOCAL_FILE_ACCESS = True\n  image_policy.ALLOWED_IMAGE_BASE_DIR = \"/data/eval_images\"\n\n  # Allow image URLs on internal or private addresses\n  image_policy.ALLOW_INTERNAL_TARGETS = True\n  ```\n\n- The metric pages document the rules and settings in full: [MultiModalFaithfulness](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/multi_modal_faithfulness/#image-loading-and-security) and [MultiModalRelevance](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/multi_modal_relevance/#image-loading-and-security).\n\n**Anthropic (`llm_factory(..., provider=\"anthropic\")`)**\n\n- `temperature` and `top_p` are no longer sent unless you set them. Older Claude models that accepted Ragas' defaults (`temperature=0.01`, `top_p=0.1`) now run at the API's default temperature, so scores can vary more between runs.\n- The default structured-output mode is now JSON mode instead of a forced tool call. Pass `mode=instructor.Mode.TOOLS` to keep the old behaviour on models that accept it.\n- An explicit `mode=` now reaches Anthropic, Google and Perplexity. It was silently ignored before.\n\n**Dependencies**\n\n- `requests` and `tenacity` are now declared as core dependencie","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}