Free OSS / CI-native LLM evals for AI agents

Local eval CLI and library. No hosted quota is included.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is MIT. Copyright line says Promptfoo 2025. README fetched 2026-10-03 says to automate checks in CI/CD. It does not say the check blocks a merge, and it does not name GitHub or GitLab. README fetched 2026-10-19 says promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, with red teaming and vulnerability scanning. Re-fetched twice 2026-10-25 (~10 s apart, plain curl, no login): GitHub blob page HTTP 200, no redirect (page chrome differs between fetches); raw LICENSE fetched twice too, HTTP 200, byte-identical: 'Copyright (c) Promptfoo 2025' / 'Permission is hereby granted, free of charge'. The file is the MIT permission text without an 'MIT License' heading. License unchanged.

UK AISI eval framework. Model provider calls are billed by those providers.

Limits: none stored.

Source · asOf 2026-10-25 · Re-fetched twice 2026-10-25 (~10 s apart, plain curl, no login): GitHub blob page HTTP 200, no redirect (page chrome differs between fetches); raw LICENSE fetched twice too, HTTP 200, byte-identical: 'MIT License' / 'Copyright (c) 2024 UK AI Security Institute'. License unchanged.

Public software-engineering benchmark instances. Download from the GitHub repo or the Hugging Face dataset princeton-nlp/SWE-bench.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card: https://huggingface.co/datasets/princeton-nlp/SWE-bench. Instance text can include upstream repository code under other licenses. Re-fetched twice 2026-10-25: https://github.com/princeton-nlp/SWE-bench/blob/main/LICENSE now answers HTTP 301 to https://github.com/SWE-bench/SWE-bench/blob/main/LICENSE (HTTP 200; the GitHub blob page changes between fetches, page chrome), so sourceUrl moved to the redirect target; the raw file https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/LICENSE was also fetched twice (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan'.

Gated public benchmark of tool-using assistant questions. Files require accepting a no-reshare condition. No license field was on the fetched card.

Limits: none stored.

Source · asOf 2026-10-25 · README fetched 2026-10-03. extra_gated_prompt says not to reshare outside a gated or private HF repo. No license key in the fetched card. Anonymous file download returned 401 earlier on the raw README path before this resolve URL. README re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter still has extra_gated_prompt 'To avoid contamination and data leakage, you agree to not reshare this dataset outside of a gated or private repository on the HF hub.' and still no license key, so license stays null.

Public cais/mmlu multiple-choice eval dataset.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. The hendrycks/test GitHub LICENSE fetched the same day is also MIT. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license:' / '- mit'.

Public MMLU-Pro eval dataset, a harder multiple-choice successor set.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. The TIGER-AI-Lab/MMLU-Pro GitHub LICENSE fetched the same day is Apache-2.0 for the code repo; this row follows the dataset card, not the code license. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: mit'.

Public function-calling eval dataset used by the Berkeley leaderboard.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: apache-2.0. The gorilla GitHub LICENSE fetched the same day is also Apache-2.0. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: apache-2.0'.

Public tau-bench agent tool-use benchmark repository.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is MIT, copyright 2024 Sierra. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2024 Sierra'.

Public terminal-bench tasks for agents that use a shell.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Apache License Version 2.0, January 2004'.

Public AgentBench multi-environment agent evaluation repository.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical), no redirect from the THUDM path: 'Apache License Version 2.0, January 2004'.

Public SWE-bench Verified subset. The dataset card fetched here does not state a license.

Limits: none stored.

Source · asOf 2026-10-25 · Hugging Face dataset API fetched 2026-10-03. cardData.license is null and gated is false. The existing swe-bench row is the MIT code license, not this subset card. License left null. Dataset API re-fetched twice 2026-10-25 (HTTP 200, byte-identical): cardData.license still null and gated false, so license stays null.

Public TheAgentCompany simulated-workplace agent benchmark.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is MIT, copyright 2024 TheAgentCompany. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2024 TheAgentCompany'.