363e497ae9
Some checks are pending
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
CI and Release / Docker / Build and test / amd64 (push) Waiting to run
CI and Release / Docker / Build and test / arm64 (push) Waiting to run
CI and Release / Docker / Publish multi-platform tags (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
* feat(benchmark): add MobileMem reproduction * fix(benchmark): declare Pillow for MobileMem images * fix(benchmark): preserve MobileMem usage and failure diagnostics * fix(benchmark): redact private keys in MobileMem diagnostics * fix(mobilemem): apply image defaults and consolidate answer receipts * fix(mobilemem): preserve response schemas during diagnostic redaction * fix(mobilemem): preserve judge parameter names in snapshots * refactor(mobilemem): share task cleanup and use persisted results --------- Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com> |
||
|---|---|---|
| .. | ||
| src/judge_mobilemem | ||
| LICENSE-CODE | ||
| pyproject.toml | ||
| README.md | ||
ReMe MobileMem Judge
Independent answer_judge Job for saved MobileMem-Omni predictions. Install with
pip install -e plugins/mobilemem-judge and enable mobilemem-judge in a separate
evaluation application/workspace, not the candidate memory application.
Inputs are question, reference, prediction, and optional evidence (a list
of original evidence values). judge_protocol defaults to membase_default, the
executable MemBase rubric used by this benchmark. native_strict selects the
separately published strict rubric. Their protocol identifiers remain distinct.
Both modes retain the official input interpolation, WRONG-first verdict parsing,
and default-zero handling of successful responses without a valid label.
response.metadata contains llm_judge_score, label, parse_default_zero,
default_judge_protocol, prompt, raw_response, and usage. Provider errors
remain failed Jobs. Judge usage is evaluation cost, not method cost.
Defaults are qwen3.8-max, temperature 0, thinking disabled and 4,096 output tokens.
JUDGE_MODEL_NAME can override the Judge model independently of the candidate.
Candidate methods and their workspaces are not dependencies of this plugin.
The benchmark runner invokes this Job in a separate application and saves its
prompt, provider output and verdict alongside the candidate prediction. Its
judge-settings.json records the validated model, generation/retry settings and
protocol identifier through a credential-free whitelist, rather than recording
only the default model name.
Sources: MemBase executable judge
and MobileMem published strict rubric.
The upstream prompt/parser code retains its MIT notice in LICENSE-CODE.