Open a project folder. ZENITH indexes git history and lays out unlinked commits on the canvas.
Select two commits. Draw a pair. Assign a Docker image and read patch difficulty.
Open the linked pair, expand hunks, then jump into File View for the full file diff.
Browse All samples and Unassigned. Linked pairs land in the sample library ready to fill.
Agents write problem statements, classify F2P/P2P, and fill missing fields with a live trace.
Create a dataset, add ready samples, and confirm readiness before you eval.
Launch supported agents in Docker and watch Pass@1/2/3 climb as samples resolve.
a3f2c91b4 Umar Ky
b7e4d9281 Umar Ky
c1a8f2e90 Umar Ky
New models ship with leaderboard scores on public suites, but those numbers say little about how an agent behaves on your repos, your diffs, and your tests. You still cannot tell which model, which agent, or which combination will hold up on the work you actually ship.
What you need is evaluation on a dataset built from your own sample work: linked commit pairs from your history, your Docker context, and your fail-to-pass and pass-to-pass tests, so you can compare agents and models on ground that is truly yours.
Select base and post commit pairs from indexed git history in a visual commit canvas.
AI fills zenbench fields (problem statement, tests, F2P/P2P classifications) with a live agent trace.
Group linked pairs into named benchmark datasets and export as JSON or JSONL.
Supported agents run in Docker sandboxes and score Pass@1, Pass@2, and Pass@3 without exposing tests during attempts.
ZENITH sits between your team and your repository, owning benchmark generation and evaluation.
Picks repos, approves samples, and reviews eval results.
Generate fields, classify F2P/P2P, and run the eval harness.
Canonical sample format with deterministic validation.
Git history, diffs, tests, and Docker execution environment.
Download ZENITH for your platform.
Docker is required for sandboxed eval runs.
Install the Python package for the API sidecar and zenbench tools.
A zenbench sample is a structured benchmark instance derived from a linked commit pair. It includes a problem statement, test command, selected test files, fail-to-pass (F2P) and pass-to-pass (P2P) classifications, Docker image, and metadata.
Fail-to-pass (F2P) tests fail at the base commit and pass at the post commit. They verify the fix. Pass-to-pass (P2P) tests pass at both commits and guard against regressions. ZENITH classifies these deterministically via zeneval.zenbench.
No. ZENITH builds and manages benchmark datasets, then runs evals through integrated harnesses like Cursor, Claude Code, Codex, and many more. You can export datasets as JSON/JSONL and use them with external evaluation pipelines.
Eval runs execute agents inside isolated Docker sandboxes so each attempt runs in a reproducible environment. Docker images are assigned per commit link and validated before generation or eval jobs start.
The desktop app stores recent folders and credentials in Tauri SQLite. Repositories, commits, links, samples, and datasets live in the Python API SQLite. The zeneval.zenbench module provides shared validation logic across both.
Generation supports OpenAI, Anthropic, Cursor, and DeepSeek. Configure one active API key and model per provider in Settings before running generation jobs.
Download ZENITH and turn your git history into structured evaluation datasets.