Free Prompt/agent eval harness for AI agents

Local eval CLI and library. No hosted quota is included.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is MIT. Copyright line says Promptfoo 2025. README fetched 2026-10-03 says to automate checks in CI/CD. It does not say the check blocks a merge, and it does not name GitHub or GitLab. README fetched 2026-10-19 says promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, with red teaming and vulnerability scanning. Re-fetched twice 2026-10-25 (~10 s apart, plain curl, no login): GitHub blob page HTTP 200, no redirect (page chrome differs between fetches); raw LICENSE fetched twice too, HTTP 200, byte-identical: 'Copyright (c) Promptfoo 2025' / 'Permission is hereby granted, free of charge'. The file is the MIT permission text without an 'MIT License' heading. License unchanged.

UK AISI eval framework. Model provider calls are billed by those providers.

Limits: none stored.

Source · asOf 2026-10-25 · Re-fetched twice 2026-10-25 (~10 s apart, plain curl, no login): GitHub blob page HTTP 200, no redirect (page chrome differs between fetches); raw LICENSE fetched twice too, HTTP 200, byte-identical: 'MIT License' / 'Copyright (c) 2024 UK AI Security Institute'. License unchanged.

Gated public benchmark of tool-using assistant questions. Files require accepting a no-reshare condition. No license field was on the fetched card.

Limits: none stored.

Source · asOf 2026-10-25 · README fetched 2026-10-03. extra_gated_prompt says not to reshare outside a gated or private HF repo. No license key in the fetched card. Anonymous file download returned 401 earlier on the raw README path before this resolve URL. README re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter still has extra_gated_prompt 'To avoid contamination and data leakage, you agree to not reshare this dataset outside of a gated or private repository on the HF hub.' and still no license key, so license stays null.

Public cais/mmlu multiple-choice eval dataset.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. The hendrycks/test GitHub LICENSE fetched the same day is also MIT. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license:' / '- mit'.

Public MMLU-Pro eval dataset, a harder multiple-choice successor set.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. The TIGER-AI-Lab/MMLU-Pro GitHub LICENSE fetched the same day is Apache-2.0 for the code repo; this row follows the dataset card, not the code license. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: mit'.

Public function-calling eval dataset used by the Berkeley leaderboard.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: apache-2.0. The gorilla GitHub LICENSE fetched the same day is also Apache-2.0. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: apache-2.0'.

Public tau-bench agent tool-use benchmark repository.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is MIT, copyright 2024 Sierra. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2024 Sierra'.

Public terminal-bench tasks for agents that use a shell.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Apache License Version 2.0, January 2004'.

Public WebArena web-agent benchmark code and task definitions.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. Self-hosted websites in the benchmark are a separate setup, not priced here. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Apache License Version 2.0, January 2004'.

Public VisualWebArena benchmark, a multimodal successor to WebArena.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 begins with a free-of-charge permission grant (MIT). Copyright 2024 Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, and Po-Yu Huang. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Copyright (c) 2024 Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, and Po-Yu Huang' / 'Permission is hereby granted, free of charge'.

Public OSWorld desktop computer-use benchmark.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Apache License Version 2.0, January 2004'.

Public Mind2Web web-agent demonstration dataset.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: cc-by-4.0. A GitHub LICENSE path returned 404. Dataset card fetched 2026-10-19 says Mind2Web is a dataset for developing and evaluating generalist agents for the web. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: cc-by-4.0'.

Public AgentBench multi-environment agent evaluation repository.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical), no redirect from the THUDM path: 'Apache License Version 2.0, January 2004'.

Public SWE-bench Verified subset. The dataset card fetched here does not state a license.

Limits: none stored.

Source · asOf 2026-10-25 · Hugging Face dataset API fetched 2026-10-03. cardData.license is null and gated is false. The existing swe-bench row is the MIT code license, not this subset card. License left null. Dataset API re-fetched twice 2026-10-25 (HTTP 200, byte-identical): cardData.license still null and gated false, so license stays null.

Public BrowserGym environment for web agents.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0, copyright 2024 ServiceNow. README fetched 2026-10-19 says it is a framework to implement, test, and evaluate web agents on BrowserGym benchmarks. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Copyright 2024 ServiceNow' / 'Licensed under the Apache License, Version 2.0'.

Public OpenAI HumanEval code-generation dataset.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. openai/human-eval GitHub LICENSE fetched the same day is also MIT. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license:' / '- mit'.

Public GSM8K grade-school math word problems.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. openai/grade-school-math GitHub LICENSE fetched the same day is also MIT. Dataset card fetched 2026-10-19 says it has been used for many benchmarks, including the Open LLM Leaderboard. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license:' / '- mit'.

Public OpenAI MRCR long-context multi-needle benchmark dataset.

Limits: none stored.

Source · asOf 2026-10-25 · Dataset card fetched 2026-10-03 says license: mit. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: mit'.

Public TheAgentCompany simulated-workplace agent benchmark.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is MIT, copyright 2024 TheAgentCompany. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2024 TheAgentCompany'.

Public multi-language SWE-bench style issue-resolution benchmark.

Limits: none stored.

Source · asOf 2026-10-25 · LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Apache License Version 2.0, January 2004'.