mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-02 02:11:29 +00:00
2 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Gergő Magyar
|
e2da8d90ce
|
test(eval): run the benchmark offline against a scripted provider (#3235)
* feat(eval): a scriptable stand-in for Anthropic and OpenAI Every defect this harness shipped last round was invisible to its own tests for one reason: the tests exercised a layer BELOW where the code runs. The usage log was never written because the proxy is a subprocess with a constructed environment. The callback could not be imported because LiteLLM loads it by path, not as a package. Failures went unrecorded because only the async hook was overridden. CI or review caught all three; no unit test could, because each called the function directly instead of driving the path that calls it. This closes that gap without spending money. It speaks the two wire protocols the harness actually depends on - Anthropic Messages, streaming and not, and OpenAI Responses - so a run can go through the real sandbox, the real CLI, the real gateway and the real usage callback with only the model faked. The runner already supports pointing at it: --base-url is the same path the free-model proxy documentation uses. Scripted rather than simulated. A test decides what the model says, which tools it asks for, and exactly what usage it reports. That last part is what makes provider-native accounting testable at all: real cache hits are not reproducible on demand, but a declared cache_read of 44,000 is. One Reply served down both protocols is also the cleanest demonstration that the same billed work is stated as a sum on one side and as a whole on the other. Tool blocks are the mechanism for artifact-producing cells. The CLI runs what it is asked to run, so a scripted Write block makes it write that file inside the sandbox for real - no model deciding anything. The end-to-end test drives the real proxy against the mock and asserts the usage log records the provider's own arithmetic through the Anthropic-shaped translation. It SKIPS here, because litellm's console script is absent in this environment, so it is unverified until CI runs it - the same footing the bubblewrap canary started on, and that one found a real bug on its first CI run. Not yet built: driving a whole sweep against this. That needs a scripted reply sequence that carries a cell to a scored artifact, which is the next step and the point of the exercise. 668 eval tests pass, 17 skipped; the two test_model_gateway.py failures are the pre-existing environmental ones. * test(eval): run a real session against the scripted provider The mock only proves something once the harness runs against it. This adds the stand-in CLI and the first integration tests that use it, so a session goes through the real code with only the model faked. tests/fixtures/fake_claude.py does what the CLI does at the two boundaries the harness depends on: it calls ANTHROPIC_BASE_URL for a turn, EXECUTES the tool blocks that come back, and prints the stream-json sequence the parent parses. Everything between - the session runner, the event-stream parse, the usage extraction, the artifact capture, the scorer - stays real. Four tests, chosen for the layers that have actually broken here: the usage a provider reported survives to the row, a scripted Write produces an artifact parse_review_output accepts, the prompt the harness meant to send is what arrived, and an upstream 529 lands as a failed session rather than a usable measurement. Writing the stand-in found two things worth keeping. The prompt arrives on STDIN under "-p --input-format text"; scanning argv for a non-flag token picks up a flag's value instead, and the prompt-fidelity test is what caught it. And three of these tests had been holding a sandbox they never applied, since no command_prefix is passed - that implied coverage which was not there, so the sandbox is gone from them and stays only in the artifact test, which needs its review directory. What these do NOT cover, checked rather than assumed: making the stand-in write in place instead of atomically still passes. On the host-unsafe backend there is no read-only mount to refuse it, so the atomic-write requirement remains a bubblewrap mount property that only the real-sandbox canary can prove. Dropping cache_read from the recorded usage does fail, so that half is genuinely pinned. 672 eval tests pass, 17 skipped; the two test_model_gateway.py failures are the environmental ones. * fix(eval): the usage adapter read a shape the callback never receives Running the gateway against the scripted provider proved the accounting merged in #3220 does not work, and the same run showed why nothing had caught it. LiteLLM does not hand a logger the upstream body. It normalises usage into its own Chat-Completions-shaped object first, so an OpenAI Responses reply reaches the callback as prompt_tokens / prompt_tokens_details.cached_tokens - never the input_tokens / input_tokens_details the shipped adapter reads. Every field came back unknown. The observed call_type is "anthropic_messages" as well, because Claude Code calls the Anthropic-shaped endpoint, so canonical_provider returned None and normalize_usage would have refused outright. Both were assumptions about a boundary I had only read about. The unit tests agreed with them because their fixture was written in the same wrong shape, so producer and consumer were consistent and both wrong - the exact failure the producer/consumer round trip exists to catch, one layer further out. Adds a LITELLM_NORMALIZED adapter for the object that actually arrives. The arithmetic is still OpenAI's - prompt_tokens is the whole, the details are subsets - so ordinary input is recovered by subtraction. The Responses adapter stays for a raw upstream body, which the mock still serves and tests directly. An unrecognised provider is still refused rather than guessed. The fixtures now carry the measured shape, and the end-to-end test asserts it through a real proxy: 48k prompt tokens with 44k cached is read back as 3k ordinary rather than as silence. 676 eval tests pass, 16 skipped, none failing. * test(eval): run a whole sweep offline, with negative controls The layers between a model turn and a promotion decision had never been exercised together. Unit tests covered each alone, and the paid runs that would have covered the composition kept dying, so the contracts BETWEEN them went unverified - which is where this harness has repeatedly shipped bugs. Drives runner.main() the way the workflow does. Real task selection, hidden oracle capture, sandbox, CLI subprocess, artifact capture, scoring against the oracle, aggregation, health guard and promotion gate. Only the model is scripted. Getting to green meant satisfying nine real contracts nothing had exercised end to end, and each failure was the harness correctly refusing bad evidence: --unsafe-no-bwrap is restricted to the paired review arms; ce_* needs a plugin carrying ce-plan, ce-work and ce-code-review; candidate_* needs an overlay; the clone needs .gitnexus/meta.json with indexedAt and lastCommit; the evidence gate needs a Skill request with a non-error result; review findings need exactly ten fields with severity in critical/high/medium/low; and the hidden labels use a DIFFERENT schema from the review output - line_start/line_end, six fields. That last one only a real run surfaces. Three negative controls, because a scorer that cannot be wrong measures nothing. A finding in the wrong place is tp=0 fp=1 fn=1 and oracle-failed, while its evidence stays VALID - being wrong is a quality result, not a broken measurement. Approving defective code is a miss with no false positive, and precision is None rather than 0, because it is undefined with no predictions. One run cannot promote: the gate says it needs three valid paired runs. A fourth control exists because a mutation demanded it. Forcing skill_was_invoked_events to return True left every other test here passing, so nothing pinned the gate that separates measuring a SKILL from measuring a model. Writing it turned up behaviour worth recording rather than assuming: a skill-not-invoked row still carries its score AND still counts toward the arm median, because aggregate() drops EXCLUDED_ERROR_KINDS and evidence_valid=False and skill-not-invoked is neither. The health guard stops the sweep, so a single-run sweep cannot promote on it, but a mixed run's median would include a cell whose skill never ran. Pinned as-is so it cannot change silently in either direction; changing it is a promotion-semantics decision, not a test fix. Two provisioning steps are stubbed and neither is harness logic: the pinned runtime mounts (no node_modules in a worktree) and the sanitized graph build (needs the gitnexus CLI at a mounted path). Containment is host-unsafe here; bubblewrap stays with the real-sandbox canary. 681 eval tests pass, 16 skipped, none failing. Runs in ~18s. * fix(eval): an uninvoked skill must not move the arm's quality median Found by the offline sweep: a skill-not-invoked row still carried its score into the arm's quality median. aggregate()'s filter dropped EXCLUDED_ERROR_KINDS and evidence_valid=False, and skill-not-invoked is neither, so an arm could be credited for a review it never performed with the skill under test - which is the one thing an arm exists to measure. Excluded from the QUALITY metrics only. Cost and duration still count that row, because the session really ran and really was billed, and the promotion gate still sees it, because it has its own vocabulary for a candidate that never loaded its skill. Two wider fixes were tried and abandoned, both because the tests said so rather than because I reasoned it out first. Reusing the health guard's evidence_failed predicate also excluded transcript-missing rows, but test_aggregate_excludes_session_error_rows_from_medians pins those as counting: that session ran, only its transcript is unverifiable. Excluding the row from `valid` outright turned a candidate whose skill never loaded from keep_incumbent into insufficient_evidence - the safety property held either way, but the decision vocabulary is promotion semantics and not mine to change on a measurement fix. Mutation-checked: putting the rows back into the quality median fails the new test. Both directions asserted, since a filter that excludes everything would also pass - a wrong-but-valid review still moves quality, because being wrong is exactly what a quality median should reflect. 682 eval tests pass, 16 skipped. * test(eval): run the offline sweep unstubbed in the job that can, and probe CLI identity Items 5 and 6 turned out to be one change. The containment (ubuntu) job already installs bubblewrap, the pinned Claude CLI, node_modules and a built GitNexus - everything the sweep's two provisioning stubs stand in for. So the stubs are not a property of the test, only of a machine that lacks those things. GITNEXUS_REQUIRE_FULL_SWEEP=1 makes the sweep run with nothing stubbed: real containment instead of --unsafe-no-bwrap, the real runtime mounts, the real sanitized graph. Set in that job, following the GITNEXUS_REQUIRE_BWRAP_CANARY pattern already there. The gate FAILS on a missing piece rather than degrading to the stubbed path, which is the point - a green tick that silently tested less is what the bubblewrap canary was written to prevent. Verified both states here: default green, and gate-on fails on this machine rather than skipping, since it cannot create user namespaces. Item 7 is an experiment, not an answer. Per-cell attribution needs an identifier that travels WITH the request, because one proxy serves the whole sweep and anything read from its environment is identical for every call. What the real CLI sends is not documented anywhere I can check, and guessing a wire format is exactly how the last three accounting bugs happened. So the probe drives the REAL pinned CLI against the mock and records the identity-bearing headers and body keys that arrive. It asserts only that a request was made; the recorded evidence is the deliverable, and the job log preserves it. Skips without CLAUDE_CANARY_BIN. Two guards caught this rather than review: the repo pins the containment job's env and its exact test list, so both had to be updated deliberately - which is the guard working, not friction. 682 eval tests pass, 17 skipped. * test(eval): make the offline sweep cross-task, so a scheduler change is checkable The sweep fixture had one task, and a single task cannot show the thing a cross-task scheduler changes: waves are per-task, so ordering, packing and a breaker spanning a task boundary are all invisible with one. A second task with its defect in a DIFFERENT file, and its own hidden labels, makes per-task routing observable. The scripted reply is now task-aware, which matters for the same reason: replying with the first task's finding scores the second task wrong. The load-bearing assertion is that each task scored against ITS OWN oracle. That is the dangerous failure mode of interleaving cells from different tasks - a mis-routed context or artifact scores one task against another's labels, and every row still looks green. Mutation-checked: pointing every cell at the first task's oracle snapshot fails it. This is the safety net the packed-scheduler wiring needs. Measured earlier against the real sweep_packed_cells, that change is worth -27% on a cold sweep and -37% weekly, with breaker fidelity holding at three injected failure positions - but it restructures a 125-line loop across ~92 names that also holds graph prefetch, reuse selection, oracle staging and the canary drop. Landing that on top of a one-task fixture would have been unverifiable, which is why this comes first and separately. 682 eval tests pass, 17 skipped. * fix(eval): commit the stand-in CLI's executable bit The file was created and chmod +x'd locally, but committed 100644 - so the mode existed only in my working tree. Any fresh checkout, CI included, gets a non-executable file and every cell dies with "required executable is not an executable regular file". Found by accident: checking out origin/main and back to compare a flaky test restored the file from the index and stripped the bit, which turned 5 green tests into 9 failures. Without that detour this would have failed on the first CI run instead. Same shape as the bugs this branch exists to catch - something that works only because of local state, breaking where the code actually runs. * fix(eval): apply code review findings Seven local reviewers and an independent cross-model pass. The headline is that a fix I added in this branch was worse than the gap it closed. Reverted the aggregate() quality-median filter. Excluding skill-not-invoked rows from the quality metrics left valid_runs and excluded_runs still counting them, so the promotion gate saw N clean runs while the median came from fewer. The dropped rows are systematically an arm's worst, so it biased toward PROMOTING - reproduced: one real run at 0.9 plus two uninvoked rows at 0.0 gave the gate 3 valid runs, zero exclusions and a 0.9 median, flipping keep_incumbent to promote. Three verdict fields compounded it: they are all() reducers still reading the wider set, so one uninvoked cell flipped a whole arm. Five reviewers found the two halves independently. Closing it honestly needs a scored-run count plus a paired-equality check in the gate, which is promotion semantics rather than an aggregation fix. The gap is now pinned by a test that states why the half-fix was reverted. Stopped forging the absence of CI. The runner refuses --unsafe-no-bwrap when CI is set because that mode runs sessions with bypassPermissions behind a boundary its own docstring calls "not a security boundary"; the sweep test deleted CI to get past it, so eval / locked pytest ran an uncontained agent sweep on the runner holding the checkout and credentials. It skips under CI instead - the containment job still runs it for real with GITNEXUS_REQUIRE_FULL_SWEEP=1. The stand-in CLI was lying in three ways. It never set is_error, so a refused write read as a completed one. It had no Skill branch at all, so honoring is_error revealed the evidence gate had been satisfied by a tool the fixture never ran - the gate was measuring the fixture, not a skill. And a reply with no usage became four zero-valued fields plus a fabricated cost, which is exactly the unknown-is-not-zero confusion the accounting it feeds exists to prevent. A provider failure also crashed the subprocess with no terminal result event. The identity probe never ran anywhere. test_mock_provider.py was in no job's file list, and the only job setting CLAUDE_CANARY_BIN runs a fixed list. My commit message claimed the next containment run would produce the answer; it would not have. Now wired in, with the CI-shape test updated to pin it. Also: the regex-miss fallback wrote a predictable name in shared /tmp through a symlink-following stage, now scoped to the test's own directory; and the canonical_provider docstring plus the callback comment still asserted a call_type branch the code no longer has. Deferred as design decisions rather than review fixes: the containment sweep uses the stand-in CLI rather than the pinned real one, the full-sweep path bypasses the gateway so native usage accounting is unexercised there, _normalize_litellm duplicates the Responses algorithm, and OPENAI_RESPONSES is now unreachable from canonical_provider. 682 eval tests pass, 17 skipped, ruff clean. * fix(eval): carry scripted tools over the Responses protocol Review round on #3235. Three real items; five more were already fixed in |
||
|
Gergő Magyar
|
18cbeb907c
|
feat(eval): record provider-native usage at the gateway instead of inferring it after translation (#3220)
* feat(eval): record provider-native usage at the gateway, not after translation
The benchmark reads token counts out of Claude Code's session output, which is
Anthropic-shaped whatever actually served the request. That holds until the
upstream is OpenAI, because the two providers do not merely name their fields
differently - they mean opposite things by them:
Anthropic: total_input = input_tokens + cache_creation + cache_read
(input_tokens is the UNCACHED remainder; cache fields ADD)
OpenAI: total_input = input_tokens
ordinary = input_tokens - cached - cache_write
(input_tokens is the WHOLE; cache fields are SUBSETS)
Adding OpenAI's three double-counts; subtracting Anthropic's under-counts. One
shared struct cannot be right for both, so the seam goes at the gateway, on the
far side of the translation: a LiteLLM callback appends each upstream request's
usage verbatim, along with the model that actually answered, the response id and
the cell it belongs to. Normalization is derived offline from that record, so the
derivation can be revisited without re-running a paid sweep.
Two rules the tests encode literally.
The native object is authoritative. The callback stores it unflattened,
unrenamed and unsummed. Reasoning tokens are kept as the decomposition of output
tokens they are, not added to them a second time.
A field nobody reported is unknown, never zero. A stored cache_read of 0 used to
mean either "the provider said zero" or "our adapter never looked" - the first
says caching is not working, the second says we cannot tell. NormalizedUsage
therefore uses None, and refuses to compute the ordinary portion when a term is
missing rather than subtracting an invented zero.
Mutation-checked three ways. Giving OpenAI Anthropic's arithmetic fails four
tests. Making unknown fall back to zero fails the unknown test. Dropping
input_tokens_details in the callback fails the end-to-end accounting test with
"assert None == 3000" - it goes unknown rather than passing with zeros, which
was the point of the exercise.
The actual model is recorded separately from the requested role because several
Claude role names map onto one upstream model here; pricing must follow what
answered. Cost is deliberately NOT stored: prices change, and tokens plus a
versioned pricing table can answer both what a past run cost and what the same
usage would cost today, without rewriting historical evidence.
The callback never raises. A cell that fails still spent money upstream, and
losing the accounting because a log write failed is the worse outcome. Failed
requests are recorded too.
No caching configuration, model, skill or promotion change: this installs the
thermometer without altering the experiment. 538 eval tests pass plus 27 gateway
tests; ruff clean. The two test_model_gateway.py failures are environmental -
litellm[proxy]'s console script is absent in this venv - and predate this branch.
* fix(eval): drop the accidentally committed .venv symlink
I symlinked eval/.venv at a sibling worktree's virtualenv to avoid rebuilding
it, and git add -A committed the symlink. .gitignore lists ".venv/" with a
trailing slash, which matches a directory and not a symlink, so nothing stopped
it.
That broke eval / containment (windows), where uv then refused to create the
environment: "failed to create directory eval\\.venv: Cannot create a file when
that file already exists". A machine-specific absolute path had no business in
the tree in the first place.
Removed, and .gitignore now also lists the bare name so the same slip cannot
repeat.
* Address PR review feedback (#3220)
Forward the usage environment into the proxy. This is the one that mattered:
the callback returns immediately when GITNEXUS_BENCH_PROVIDER_USAGE is absent,
the proxy runs as its own process, and Popen(env=...) REPLACES the parent
environment rather than extending it. The gateway's allowlist carried the
OpenAI and master keys and nothing else, so the callback loaded, found no
destination, and silently recorded nothing on every request. The accounting
looked configured and measured nothing at all.
My tests could not see it. They set the variable in-process and called the
logger directly, so none of them ever crossed the subprocess boundary the
feature actually runs behind. The new test drives OpenAIGateway.__enter__ with
Popen captured and asserts each variable reaches the child - and that the
result is still an allowlist rather than the inherited parent environment,
since forwarding by name is what keeps the credential boundary explicit.
Resolve the provider label into an adapter key. The callback recorded
LiteLLM's custom_llm_provider, which is "openai", while the adapter table is
keyed "openai-responses" - so nothing the logger wrote could have been
normalized. The end-to-end test hid this by passing OPENAI_RESPONSES by hand
instead of using the provider the log recorded; it now uses the logged value,
which is what makes the mismatch visible.
The label alone cannot pick an adapter: LiteLLM reports "openai" for Chat
Completions as well, and the two report usage differently. canonical_provider
combines the label with the call type and returns None when it cannot resolve
one, so normalize_usage refuses rather than guessing token semantics. Both are
stored - provider_label is what LiteLLM said, provider is the adapter key.
The shared env-var names moved into provider_usage.py so model_gateway can
import them without importing litellm, which only the in-proxy callback needs.
Mutation-checked. Removing the forwarding loop fails the gateway test; using
the raw label as the adapter key fails two.
656 eval tests pass, ruff clean. The two test_model_gateway.py failures are the
environmental ones - litellm[proxy]'s console script is absent here, which is
also why the new test patches the argv builder to reach Popen at all.
* fix(eval): stop recording a cell id the proxy cannot know
Setting out to build the correlation this PR was missing - cell usage as the
sum of its upstream requests - turned up that the field it would have been
built on cannot hold what its name claims.
attach_openai_gateway wraps the whole sweep (runner.py:2122), so ONE proxy
serves every cell, and its environment is fixed for that process's lifetime.
Cells run concurrently under --workers and interleave requests through it. A
cell id forwarded at launch is therefore the same constant on every event the
callback ever writes - not an attribution, just a label that looks like one.
Worse than absent, because a reader would trust it.
So GITNEXUS_BENCH_CELL_ID is gone rather than left to be wired up later. What
remains is honest about its scope: sweep_id is genuinely sweep-wide, and
session_id is the per-request half - the only thing that can attribute a
request to a cell, since anything read from the environment is shared by all of
them. It is recorded even when the provider supplies nothing, because knowing
attribution is unavailable is itself a fact about the run.
Pinned by a test asserting the forwarded set contains no per-cell variable, so
a later change does not reintroduce one and quietly stamp a single value across
concurrent cells.
What this leaves open, stated plainly: per-cell attribution is NOT built, and
cannot be until a per-request identifier is available. Whether Claude Code
propagates a session identifier through the proxy is unverified - determining
it needs a real session against the gateway, which is a paid run. Sweep-level
totals and per-request cache ratios do not need it, and those are what the
caching question actually turns on.
658 eval tests pass, ruff clean; the two test_model_gateway.py failures remain
environmental.
* fix(eval): keep the usage callback importable the way LiteLLM loads it
CI caught a regression I introduced: "ImportError: Could not import handler
from provider_usage_callback", and the proxy exited before becoming ready.
Moving the shared constants into provider_usage.py, I imported them from the
callback with "from .provider_usage import ...". But LiteLLM resolves a dotted
callback through spec_from_file_location against the config directory, so the
copied file runs as a top-level module with no parent package and no sys.path
entry - the relative import raises and the gateway never starts. The module's
own docstring says it is deliberately self-contained for exactly this reason,
and I broke that invariant while tidying.
The in-package tests could not see it. They import
workflow_bench.litellm_usage_callback, where the relative import resolves
fine; the failure only exists on the path where the file is copied and loaded
standalone.
The callback carries its own literals again. Two tests keep that honest: one
loads the copied file the way LiteLLM does - by path, as a top-level module -
so an import that only works in-package fails there, and one asserts the
copied constants and the provider resolver still agree with the canonical
copies in provider_usage.py, so the deliberate duplication cannot drift
silently.
Mutation-checked: restoring the relative import reproduces CI's exact error.
660 eval tests pass locally; the two remaining test_model_gateway.py failures
are the environmental ones (litellm[proxy]'s console script is absent here,
which is also why this never reproduced locally).
* test(eval): import the installed callback instead of grepping it
Two review findings on the same weakness, both correct.
The install test asserted "class ProviderUsageLogger" appeared in the copied
file's text. That passes whenever the string is present, including when the
module cannot load at all - which is precisely how a package-relative import
got through review here and took the proxy down. It now loads the copy the way
LiteLLM does, by path as a top-level module, and checks the handler instance
the config actually names.
The gateway-forwarding test built its work directory with tempfile.mkdtemp(),
which nothing removed, so every run left the generated config and the copied
callback behind in the system temp directory. It uses the pytest-managed
tmp_path fixture like its neighbours.
660 eval tests pass; the two test_model_gateway.py failures are the
environmental ones.
* fix(eval): record failures on the synchronous callback path too
ProviderUsageLogger overrode both async hooks and the sync SUCCESS hook, but
not the sync failure hook. On that path failures fell through to CustomLogger's
base implementation and were never appended - so a sweep recorded its
successes and quietly understated what it spent, since a failed request is
billed all the same. That contradicts the module's own stated reason for
handling failures at all.
The failure test could not have caught it: it called _append directly, which
exercises neither public hook. Both failure tests now drive the hooks LiteLLM
actually calls, and a new one walks all four - sync and async, success and
failure - asserting each records in order. Removing the sync failure hook fails
both.
661 eval tests pass; the two test_model_gateway.py failures remain
environmental.
---------
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
|