Storing a baseline used to prune every other cache entry, so gate runs in
concurrent worktrees kept evicting each other's baselines and forcing full
recomputes: this bit six times across two nights of benchmarking. The store
now writes alongside existing entries and evicts only the oldest beyond
eight, keyed as before by merge-base and environment fingerprint, so
parallel worktrees' baselines simply coexist
Prompt Security drops tool and function rows unless check_tool_results
is on, so it now reports scan-only support from that setting and the
registry refuses the pairing at boot. Pairing scan_only_tool_results
with skip_tool_message_in_guardrail excludes every message, so guardrail
initialization now rejects that combination too.
prisma resolves its prisma-client-py generator through a plain /bin/sh PATH
lookup, never through the interpreter that ran prisma generate, so the gate's
generate step landed the client in whatever venv the caller had on PATH: the
owned env never received one, every gate run regenerated, the caller's venv
was mutated instead, and any invocation without a venv on PATH (the rewritten
publisher workflow) failed outright
The generate now runs with the target interpreter's bin directory pinned to
the front of the child PATH. The prisma schema joins the environment
fingerprint so clientless counts recorded before this commit can never be
compared against clientful ones, a cold provision announces itself on stderr
instead of sitting silent for two minutes, and the CI gate step reuses the
job's prisma binary cache
Under scan_only_tool_results, legacy OpenAI function-role messages now count as tool results, and duplicate names among guardrail-returned tools keep only the first occurrence. CustomGuardrail.structured_messages_cover_full_request lets CrowdStrike AIDR declare that its writeback already rebuilds the whole conversation, so handlers install it as-is instead of merging it into the full message list a second time and duplicating out-of-scope rows. Lint budget ceilings ratchet down to match the tree
Replaces the hand-rolled recording double with AsyncMock so the awaited team ids
come from await_args_list instead of a mutated list, and annotates the response
factory now that SearchToolInfoResponse is imported at module level.
The aggregated route was missing from LiteLLMRoutes.self_managed_routes
while its paginated sibling /user/daily/activity was listed, so auth
rejected every internal user with a 401 before the handler ran. That
route backs the default "Your Usage" view in the dashboard, which left
the main Usage page broken for non-admin users.
The handler already self-scopes: it checks admin view first, then falls
back to require_caller_user_id_for_non_admin, defaults a missing user_id
to the caller's own, and returns 403 when a non-admin asks for someone
else's data. Listing the route restores reachability without widening
what a caller can read.
check_route_access matches exactly (plus explicit wildcards), so the
parent entry never covered the /aggregated sub-path.
Every Admin UI session key is stamped with the reserved team id
`litellm-dashboard`, which never has a row in LiteLLM_TeamTable, so the
team lookup in _filter_visible_search_tools raised 404 and the endpoint
returned 500 for every non-admin dashboard session.
Skip the lookup for that sentinel and scope the caller by its key-level
allowlist alone, matching how MCP and agent permission checks already
treat it. A real team id is still resolved, and a genuine lookup failure
now surfaces with its own status instead of being masked as a 500.
* fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day
* fix(proxy): gate the current-UTC-day extension behind an opt-in param sent by the cost optimization dashboard
* fix(ui): label cost optimization savings dates as UTC days
get_deployment_credentials_with_provider dropped s3_region_name,
s3_encryption_key_id, and aws_batch_role_arn because
CredentialLiteLLMParams never declared them, and it never returned the
deployment's model, so proxy batch creation against Bedrock failed with
"LiteLLM doesn't support custom_llm_provider=bedrock for 'create_batch'"
or "AWS IAM role ARN is required" (#25104)
Provider-only file and batch calls keep their no-model contract:
get_team_provider_credentials strips the model key so a provider-scoped
request is not pinned to an arbitrary matching deployment
An adversarial review of the previous commit found annotations that
described what the code wished were true rather than what flows through.
A false annotation is worse than the Any it replaced, since it launders a
wrong assumption past the type checker.
- purview: `_resolve_user_id` claimed every request-body value was a
Mapping, contradicting `_resolve_trusted_user_id` one method over, which
types the same argument `Mapping[str, object]`. `_should_block` claimed
every Graph response value was a sequence of str->str mappings and was
not assignable from its own producer's return type.
- cato: `_CatoAnalyzeResponse.required_action` was required and
non-nullable while the API returns null, as seven fixtures in the
guardrail's own suite assert. `analysis_result` had the same problem.
The streaming hook narrowed an override parameter below what
`ProxyLogging` actually passes it.
- marketplace: `_PluginRecord.manifest_json` was `str` against a nullable
column. Making it honest surfaced a latent crash, covered below.
- ownership: two functions took an attribute Protocol while their own
bodies branch on `isinstance(response, dict)`, which no Protocol can
satisfy.
- openapi generator: `paths` claimed every path-item value was an
operation, though path items also carry `parameters`, `summary` and
`$ref`.
- custom openapi spec: a TypedDict asserted a shape that the function
returns raw Pydantic sub-schemas out of. Reverted to Any, which is
imprecise but not false.
`get_marketplace` did an unguarded `json.loads` on the nullable
`manifest_json` inside an `except json.JSONDecodeError`, which cannot
catch the TypeError a NULL raises, so one NULL row 500s the endpoint. It
now skips the plugin like the file's other two read sites already do, with
a regression test that fails without the guard.
Where honesty cost precision, precision lost. `_should_block` went back to
its original signature entirely: the narrowing needed to type it turned a
fail-closed DLP control fail-open, because the TypeError it used to raise
on a malformed response reached `except Exception` and became a 400.
The gate previously measured whatever environment the caller happened to
have. Locally that is the fat bootstrap venv (--extra proxy pulls in
fastapi-sso, whose type info flips a reportUnnecessaryIsInstance
diagnostic in ui_sso.py), while CI's publisher venv only has the
proxy-dev and e2e-dev groups, so identical trees measured 866 locally vs
865 in CI and every local gate run breached by a phantom +1
scripts/type_check_gate.py now provisions .venv-typecheck itself: a
frozen uv sync of the canonical proxy-dev and e2e-dev groups, the
interpreter pinned to pyrightconfig.json's pythonVersion, plus the
generated Prisma client. Every measurement pass is pinned to that env
with --pythonpath, because basedpyright auto-detects a .venv in the
project root and that auto-detection beats both PATH order and
VIRTUAL_ENV, so the CLI flag is the only pin that actually works. The
dependency-group set is folded into the environment fingerprint, so
artifacts or caches recorded under a different group set never match
and the gate falls back to computing base counts locally instead of
comparing mismatched environments
The publisher workflow drops its own install and prisma steps and lets
the script build the measurement env, and the node heap for the
full-tree pass drops from 12GB to 8GB (peak RSS measured at 5.4GB)
* fix(auto-router): accept every reminder marker pair a harness emits
reminder_markers held one (open, close) pair, so a harness that wraps
injected context differently per agent type only got the slice of traffic
using the configured envelope stripped. Every other agent type kept hitting
the original bug: its reminder-only turn never stripped to empty, won
"newest human ask", and the harness blob got classified in place of the
real question, choosing the tier and therefore the spend.
The field now takes a list of ReminderMarkerPair, following the
KeywordTierRule pattern already in this file so each pair validates itself
and errors point at reminder_markers.N.close rather than a bare index.
Blocks from different pairs can nest, which the gap construction could not
handle: resuming the kept text at an inner block's end walks back inside
the enclosing block and leaks its remainder. Running the block ends through
a maximum collapses nested and overlapping spans without a separate merge
pass, and stays linear in block count, which a fold over a growing tuple
of merged spans would not.
A single pair's ends already increase, so the maximum is the identity and
the default path is byte-identical: verified against the shipped function
over 200k generated inputs, and every existing reminder test passes
unchanged. The prior single-pair config shape is rejected loudly at
startup and at /model/new rather than silently stripping nothing.
* docs(auto-router): document reminder_markers in the complexity router README
* chore(ui): regenerate dashboard API types for the reminder_markers shape
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
list_user_batches parsed each stored batch blob and returned it as-is, so any
row whose blob still carried raw provider file ids (for example a batch that
reached a terminal state through the cost poller, or rows written before
output registration existed) leaked raw output_file_id and error_file_id
values that clients cannot fetch through the proxy. The list path now runs
each row through ensure_batch_response_managed_file_ids, which swaps in
existing managed ids and registers missing ones under the batch owner's
identity, matching what GET /batches/{id} already does
Since #35491, register_model records every registration in the process-global
_runtime_registered_model_cost ledger, and every cost map swap replays that
ledger on top of the freshly adopted map. Under pytest-xdist, any earlier test
in the same worker that registered gpt-3.5-turbo leaked into
TestPriceDataReloadIntegration::test_distributed_reload_check_function: the
replay ballooned its sparse mocked entry into a full ModelInfo dict and failed
the exact-equality assert, breaking the proxy-infra shard whenever loadscope
happened to co-schedule such a test first (reruns cannot help since the
pollution is process-wide)
The autouse isolate_litellm_state fixture now snapshots the ledger before each
test and restores it in place on teardown, so no test's registrations outlive
it. A regression pair in test_conftest_isolation.py asserts the rollback
When the batch cost poller found a batch in a terminal failed, expired, or
cancelled state it wrote the provider response straight to the managed object
table, so the stored blob kept raw provider file ids and a raw batch id. Since
the row is final after batch_processed=True and the read paths only resolve
existing managed ids, every later GET /batches/{id} and GET /batches leaked
raw provider output and error file ids that clients cannot fetch through the
proxy. The terminal branch now normalizes the response with
ensure_batch_response_managed_file_ids before persisting, minting managed ids
under the batch owner's identity
POST /batches/{id}/cancel had the same gap: it called update_batch_in_database
without the caller's auth context, so a cancel response that already carried
provider file ids could never mint managed ids. The endpoint now forwards
user_api_key_dict
OpenAI emits a reasoning output item on every reasoning turn, but only emits
reasoning_summary_text deltas when a summary was requested and actually
produced. The Anthropic /v1/messages Responses stream adapter opened the
thinking content block eagerly on response.output_item.added, so a summary-less
reasoning item surfaced as {"type": "thinking", "thinking": ""}. Clients persist
that in their session transcript and replay it on the next turn; an Anthropic
model then rejects the request with "each thinking block must contain thinking",
which is what users hit when a resumed session falls back to the default
Anthropic model.
Open the thinking block on the first non-empty summary delta instead, and only
emit content_block_stop for items that actually have an open block.
Clients send tool_choice as {"type": "auto"} (Cursor on chat completions,
Claude Code's Anthropic tool_choice shape). validate_chat_completion_tool_choice
recognized that shape but returned it verbatim, and the chat -> Responses API
bridge only normalized {"type": "function"}, so the wrapper reached OpenAI and
the whole call failed with:
Invalid value: 'auto'. Supported values are: 'code_interpreter', ...,
'web_search_preview', ... (param: tool_choice.type)
That broke every tool call, web search included, on responses-mode models.
Unwrap {"type": "auto"|"none"|"required"} to the bare string at both layers:
the chat completions validation boundary where the shape is first accepted,
and the Responses API bridge that owns the Responses tool_choice contract.
No OpenAI surface accepts the object form for these values, so the previous
passthrough only deferred the 400 to the provider.
Resolves conflicts from the LIT010/LIT011 Final-enforcement lint pass
landing on litellm_internal_staging after this branch diverged. Keeps
this PR's behavior changes (float support in _UrlEncodableParams,
in-place results truncation + header stashing in
transform_search_response) and adopts the upstream Final annotations
and updated _TINYFISH_RESULT_CAP comment.
update_batch_in_database now fetches the batch row by unified_object_id
when the caller omits db_batch_object, so the cancel endpoint attributes
newly registered output and error files to the batch owner and returns
unified managed ids instead of raw provider ids. Idempotent cancels that
do not change the stored status also skip the redundant DB write now.
Repair two pre-existing mock tests in test_openai_batches_endpoint.py
that asserted values inside lazy percent-format log strings, and give
the cancel test's prisma mock an awaitable find_first.
Gate the OpenAI handler's tools forwarding behind scan_only_tool_results,
matching the Anthropic handler, so a tool-results-only scan can no longer
evaluate or rewrite trusted function definitions.
When a guardrail returns a replacement structured_messages list, substitute
the returned messages back into the positions their scoped originals came
from instead of installing the scoped list as the whole conversation, so
out-of-scope messages (system prompt, prior turns) survive redaction on
both the OpenAI and Anthropic paths.
