ai-benchmarking

Benchmarking the gap between AI agent hype and architecture. Three agent archetypes, 73-point performance spread, stress testing, network resilience, and ensemble coordination analysis with statistical validation.

FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

Eval framework. Define correct, test against it, get results.

LLM Benchmark

AI model hub, benchmark arena, and multi-provider playground built with Next.js, live metadata providers, and a fast REST cache layer.

A terminal widget that allows you to keep track of Codex usage against your current quota, command sessions and more ...

HiddenBench: Benchmark for evaluating collective reasoning in multi-agent LLM systems

AI agent evaluation framework for multi-participant coordination tasks. Built with LangGraph, custom MCP tools, and LLM-as-a-Judge evaluation. MSc dissertation project (University of Edinburgh, 2025).

ORBIT - Operational Research and Benchmarking Interface for Transformers

🛡️ Artemis: A high-fidelity OpenEnv benchmark for evaluating AI agents in realistic SOC triage and cybersecurity incident response scenarios.

An evidence-hostile, container-isolated benchmark and behavior analysis platform for long-horizon AI software agents.

Open-source, local-first evaluation infrastructure for applied AI systems, built for developer and agent workflows.

An open public-interest project exploring compassionate, responsible AI stewardship, ethical AI literacy, human agency, and long-term continuity.

SWE-bench for your private repo. Benchmark AI coding agents in isolated Docker.

Uncertainty & Confidence Management (UCM): A healthcare AI benchmark suite for uncertainty recognition, justification boundaries, confidence calibration, proportionate action, and reassessment.

A comprehensive benchmarking platform for CPT, ICD-10, and HCPCS coding questions. Identifies the most reliable models for healthcare applications. Evaluates multiple AI models on medical coding expertise through iterative consensus-building.

Comprehensive multi-IDE AI model benchmarking framework supporting Cursor, Windsurf, VSCode, and other IDEs with automated testing and performance comparison capabilities

A benchmarking framework for designing and evaluating multi-agent AI systems. Implements structured task decomposition (Map-Reduce/Fan-out), containerized execution, and deterministic verification for complex AI orchestration

Reproducible AI benchmarks, experiments, reusable jobs, API examples and evidence-backed reports for Agent Control.

Which programming language is best for AI coding agents? Benchmarking 13 languages with Claude Code.