scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-08 22:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 78

refactor(shadow_eval): tighten the judge cap comment and type the test helper

Browse source
This commit is contained in:
moe-berri 2026-09-04 16:21:13 -07:00
parent a2f926eb8f
commit 2f5bfae1a6
2 changed files with 8 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
litellm/integrations/shadow_eval_logger.py
View file

@ -60,12 +60,9 @@ _MAX_CONCURRENT_SHADOW_TASKS: Final = 16
_MAX_JUDGE_RESPONSE_CHARS: Final = 8_000
_MAX_JUDGE_PROMPT_CHARS: Final = 24_000
# The judge answers with a small JSON object, but the cap covers reasoning tokens too. A
# judge_model deployment configured with an elevated reasoning_effort or thinking budget
# (a realistic pick: an admin's best reasoning model doubling as the judge) spends most or
# all of a tight cap on that reasoning, invisibly to this call, and the reply arrives empty
# or truncated mid-object, which the attempt records as an unparseable verdict. Headroom is
# free: max_tokens is a ceiling, and only generated tokens bill.
# The judge answers with a small JSON object, but the cap covers reasoning tokens too: a
# judge deployment carrying an elevated reasoning_effort spends a tight cap before it ever
# answers, and the truncated reply is recorded as an unparseable verdict.
JUDGE_MAX_OUTPUT_TOKENS: Final = 4096
_MAX_ERROR_CHARS: Final = 500

10
tests/test_litellm/integrations/test_shadow_eval_logger.py
View file

@ -120,12 +120,12 @@ def _router(
return router
def _reasoning_judge_router(reasoning_tokens, verdict='{"preference": "A", "confidence": 0.9}'):
def _reasoning_judge_router(
reasoning_tokens: int, verdict: str = '{"preference": "A", "confidence": 0.9}'
) -> MagicMock:
"""A router whose judge arm reasons before it answers, the way a deployment carrying an
elevated reasoning_effort does. Reasoning is billed against the caller's own max_tokens
and the reply is cut off at that cap, so a cap that does not clear the reasoning budget
yields a truncated verdict or no verdict at all. One character stands in for one token,
which is what makes the cap the thing under test."""
elevated reasoning_effort does: reasoning bills against the caller's own max_tokens and
the reply is cut off at that cap. One character stands in for one token."""
router = MagicMock()
router.model_group_alias = {}
router.get_model_list = MagicMock(return_value=[{"litellm_params": {"model": "openai/gpt-4o-mini"}}])