EvalCore is snapshot testing for AI behavior. It records how your LLM app or agent behaves and replays this behavior in CI on every change, ensuring offline, deterministic testing with no network, API keys, or cost.
Key benefits include:
- Record & Replay in CI: Run once to record real model interactions to a local SQLite cassette, then replay in CI for every PR—no network, API keys, or flaky judges, ensuring identical inputs produce identical verdicts.
- Simple Configuration: Define targets, datasets, and scorers in a YAML file (e.g., evals.yaml), with no SDK or test harness required—works with any language or app that speaks HTTP or shell.
- Free & Deterministic Testing: Cassettes are free to commit and replay; offline, keyless, and cost-free, with identical inputs guaranteeing consistent verdicts across environments.
- Multi-Model Compatibility: Supports OpenAI, vLLM, Ollama, OpenAI-compatible gateways, REST APIs, shell commands, and OTel/OpenInference traces for versatile AI evaluation.
Perfect for developers building LLM apps, agents, or AI-powered products who need to ensure reliability and detect regressions in AI behavior before users notice.
