* fix(auto-router): accept every reminder marker pair a harness emits
reminder_markers held one (open, close) pair, so a harness that wraps
injected context differently per agent type only got the slice of traffic
using the configured envelope stripped. Every other agent type kept hitting
the original bug: its reminder-only turn never stripped to empty, won
"newest human ask", and the harness blob got classified in place of the
real question, choosing the tier and therefore the spend.
The field now takes a list of ReminderMarkerPair, following the
KeywordTierRule pattern already in this file so each pair validates itself
and errors point at reminder_markers.N.close rather than a bare index.
Blocks from different pairs can nest, which the gap construction could not
handle: resuming the kept text at an inner block's end walks back inside
the enclosing block and leaks its remainder. Running the block ends through
a maximum collapses nested and overlapping spans without a separate merge
pass, and stays linear in block count, which a fold over a growing tuple
of merged spans would not.
A single pair's ends already increase, so the maximum is the identity and
the default path is byte-identical: verified against the shipped function
over 200k generated inputs, and every existing reminder test passes
unchanged. The prior single-pair config shape is rejected loudly at
startup and at /model/new rather than silently stripping nothing.
* docs(auto-router): document reminder_markers in the complexity router README
* chore(ui): regenerate dashboard API types for the reminder_markers shape
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
list_user_batches parsed each stored batch blob and returned it as-is, so any
row whose blob still carried raw provider file ids (for example a batch that
reached a terminal state through the cost poller, or rows written before
output registration existed) leaked raw output_file_id and error_file_id
values that clients cannot fetch through the proxy. The list path now runs
each row through ensure_batch_response_managed_file_ids, which swaps in
existing managed ids and registers missing ones under the batch owner's
identity, matching what GET /batches/{id} already does
Since #35491, register_model records every registration in the process-global
_runtime_registered_model_cost ledger, and every cost map swap replays that
ledger on top of the freshly adopted map. Under pytest-xdist, any earlier test
in the same worker that registered gpt-3.5-turbo leaked into
TestPriceDataReloadIntegration::test_distributed_reload_check_function: the
replay ballooned its sparse mocked entry into a full ModelInfo dict and failed
the exact-equality assert, breaking the proxy-infra shard whenever loadscope
happened to co-schedule such a test first (reruns cannot help since the
pollution is process-wide)
The autouse isolate_litellm_state fixture now snapshots the ledger before each
test and restores it in place on teardown, so no test's registrations outlive
it. A regression pair in test_conftest_isolation.py asserts the rollback
When the batch cost poller found a batch in a terminal failed, expired, or
cancelled state it wrote the provider response straight to the managed object
table, so the stored blob kept raw provider file ids and a raw batch id. Since
the row is final after batch_processed=True and the read paths only resolve
existing managed ids, every later GET /batches/{id} and GET /batches leaked
raw provider output and error file ids that clients cannot fetch through the
proxy. The terminal branch now normalizes the response with
ensure_batch_response_managed_file_ids before persisting, minting managed ids
under the batch owner's identity
POST /batches/{id}/cancel had the same gap: it called update_batch_in_database
without the caller's auth context, so a cancel response that already carried
provider file ids could never mint managed ids. The endpoint now forwards
user_api_key_dict
OpenAI emits a reasoning output item on every reasoning turn, but only emits
reasoning_summary_text deltas when a summary was requested and actually
produced. The Anthropic /v1/messages Responses stream adapter opened the
thinking content block eagerly on response.output_item.added, so a summary-less
reasoning item surfaced as {"type": "thinking", "thinking": ""}. Clients persist
that in their session transcript and replay it on the next turn; an Anthropic
model then rejects the request with "each thinking block must contain thinking",
which is what users hit when a resumed session falls back to the default
Anthropic model.
Open the thinking block on the first non-empty summary delta instead, and only
emit content_block_stop for items that actually have an open block.
Clients send tool_choice as {"type": "auto"} (Cursor on chat completions,
Claude Code's Anthropic tool_choice shape). validate_chat_completion_tool_choice
recognized that shape but returned it verbatim, and the chat -> Responses API
bridge only normalized {"type": "function"}, so the wrapper reached OpenAI and
the whole call failed with:
Invalid value: 'auto'. Supported values are: 'code_interpreter', ...,
'web_search_preview', ... (param: tool_choice.type)
That broke every tool call, web search included, on responses-mode models.
Unwrap {"type": "auto"|"none"|"required"} to the bare string at both layers:
the chat completions validation boundary where the shape is first accepted,
and the Responses API bridge that owns the Responses tool_choice contract.
No OpenAI surface accepts the object form for these values, so the previous
passthrough only deferred the 400 to the provider.
Resolves conflicts from the LIT010/LIT011 Final-enforcement lint pass
landing on litellm_internal_staging after this branch diverged. Keeps
this PR's behavior changes (float support in _UrlEncodableParams,
in-place results truncation + header stashing in
transform_search_response) and adopts the upstream Final annotations
and updated _TINYFISH_RESULT_CAP comment.
update_batch_in_database now fetches the batch row by unified_object_id
when the caller omits db_batch_object, so the cancel endpoint attributes
newly registered output and error files to the batch owner and returns
unified managed ids instead of raw provider ids. Idempotent cancels that
do not change the stored status also skip the redundant DB write now.
Repair two pre-existing mock tests in test_openai_batches_endpoint.py
that asserted values inside lazy percent-format log strings, and give
the cancel test's prisma mock an awaitable find_first.
Gate the OpenAI handler's tools forwarding behind scan_only_tool_results,
matching the Anthropic handler, so a tool-results-only scan can no longer
evaluate or rewrite trusted function definitions.
When a guardrail returns a replacement structured_messages list, substitute
the returned messages back into the positions their scoped originals came
from instead of installing the scoped list as the whole conversation, so
out-of-scope messages (system prompt, prior turns) survive redaction on
both the OpenAI and Anthropic paths.
The skip warning interpolated the full pydantic ValidationError, whose
string embeds input_value with the rejected row's contents. Managed-file
rows carry a caller-supplied filename, so a malformed row copied that
into operational logs.
Log the error locations, types, and messages via errors() with input,
url, and context excluded, keeping the field-level diagnostics without
the values. Non-validation failures fall back to the exception type.
get_user_created_file_ids validated every row's file_object without a
guard, so a single row failing OpenAIFileObject validation raised
ValidationError and turned the whole GET /v1/files response into a 500.
#35365 covered the null case only, leaving malformed or partial rows
able to take the entire listing down.
Rows now parse through a helper that returns None on failure and logs a
warning, matching how list_user_batches already tolerates rows it cannot
parse, so one bad row costs its own entry instead of the caller's whole
listing. Null rows stay silent since the batch cost poller registers
those legitimately.
Refs #35361
Terminal batch retrieve could return the raw provider output_file_id, which
skips managed-file ownership checks on /v1/files/{id}/content and lets any key
on the proxy download another user's batch output.
Retrieve now registers the missing managed-file row before responding, and
attributes ownership to the durable batch owner rather than the retrieving
caller, so output and error ids always come back as unified managed ids.
Fixes#33989
* fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion
* fix: refresh models_by_provider in place so captured references survive reloads
A connection test that redirects the destination already leaves the configured
credentials behind. It kept litellm_credential_name, which names the same stored
secrets and is resolved further down the call, so the reference is now dropped
with them. A request that sets no connection fields of its own is unaffected,
which is how the Admin UI tests a configured model.
The proxy-wide opt-in that already governs callers supplying their own
connection parameters now also governs whether a connection test may pair a
request-supplied endpoint with the configured deployment's credentials. Off by
default, which keeps configured credentials scoped to the endpoint the
configuration names; on, the previous merge behaviour is available unchanged.
A request that supplies its own connection fields describes a connection of its
own, so the configured deployment's credentials are no longer merged underneath
it. Anything the request leaves unset still comes from the configuration, so
naming a configured model and testing it as configured is unchanged, and adding
a second deployment for an already-configured name works as before.
Replaces the earlier outright rejection, which also refused requests that
supplied a complete connection of their own.
When a connection test names a model that resolves to a configured deployment,
that deployment's routing and credential parameters are authoritative. A request
supplying a complete connection of its own is unaffected.
BREAKING CHANGE: /health/test_connection no longer lets a request replace the
routing or credential parameters of a configured model it names. Supply the full
connection parameters instead of naming a configured model.
Multipart callers express nested metadata as flat bracket-notation keys, which
reach the request-body check as literal keys rather than as a metadata dict.
The check now rebuilds them with the same helper the endpoints use, so both
encodings are handled identically and cannot drift apart.
BREAKING CHANGE: a multipart field such as `litellm_metadata[api_base]` is now
subject to the same request-body parameter rules as its JSON equivalent. Set
`general_settings.allow_client_side_credentials`, or the deployment's
`configurable_clientside_auth_params`, to keep passing these.
The URL-destination check previously ran over request-body fields only. The
per-field logic moves into reject_url_valued_destination(field, value) so a
deployment name resolved from the request path runs the same check against the
same admin allowlist.
BREAKING CHANGE: a deployment name supplied in the request path that parses as
an http/https destination is now refused. Add the host to
`provider_url_destination_allowed_hosts` in litellm_settings to keep it working.
* feat(pre-commit): save full lint output to a per-worktree log file
* docs(claude): point agents at the pre-commit log instead of rerunning
* fix(pre-commit): warn when the log cannot be created or fully written
An SSE stream that cannot be positively identified as Anthropic (no
parseable message_start event) now blocks instead of passing through
unscanned, closing the bypass where any raw-SSE backend skipped tool
permission checks entirely. Buffered chunks are joined back into one
stream before parsing, so events split across network chunk boundaries
assemble correctly instead of being silently dropped. Rewrite mode now
resets finish_reason to stop when no tool call survives, so the
re-encoded Anthropic stream reports stop_reason end_turn and clients do
not wait for a tool result that never comes
A cached HTTPHandler hands its raw httpx.Client out to consumers that keep it
for the process lifetime. When the shared client cache expires the entry on its
TTL or evicts it under the 200-entry cap, nothing references the handler, so it
is collected and its finalizer closed the client those consumers still hold.
Langfuse ingestion then failed silently on the SDK's background flush thread
until the process restarted.
A finalizer running proves only that nothing references the handler; it proves
nothing about the client. Both handlers now close the client during finalization
only when they built it and are still its sole referrer, so an unshared client is
still released promptly and a handed-out one is left alone. That keeps the
pooled-socket reclamation the finalizer was providing, which measures identical
to base over 2000 handler create-and-drop cycles.
Explicit close() stays, now gated on _owns_client so the wrapper never closes a
caller-injected client, and __aexit__ routes through it.
LangFuseLogger also keeps a reference to the handler whose client it hands the
SDK. Previously that handler was a local that went out of scope immediately,
leaving the client reachable only from the SDK. It still shares the cached
client, so no extra clients are created per logger.
test_reapply_runtime_registrations_replays_register_model_overrides asserts that
a fetched catalog value survives the replay for a key an operator override does
not mention. Any Router still alive in the process re-asserts its own deployments
first, so a router serving openai/gpt-4o writes its model_info over that catalog
value and the assertion reads the router's number instead. Routers built by
earlier tests stay in the weak set until they are collected, which made the test
depend on collection timing and fail intermittently in shards that run the router
tests alongside it.
The live-router rebuild is covered in test_router_model_cost_isolation.py, so
this test now runs with the replay callback unset and exercises the recorded
registrations it is about.
The redaction filter was attached to the handler shared by litellm's own
loggers, so it only covered records litellm emits. A litellm value can also
reach a log record through a dependency logging on its own logger, and those
records never pass through a litellm handler.
Attach the filter to each dependency logger that can carry one. The filter goes
on the emitting logger rather than on the root logger or a root handler, since
Logger.handle applies the emitting logger's filters before any handler runs, so
every downstream handler is covered regardless of who owns it.
The gate required a litellm. prefix on get_secret and get_secret_str, so any
module importing either function directly bypassed it: 335 environment variables
read under litellm/ were invisible to it. The three patterns collapse into one
with the prefix optional, a negative lookbehind so attribute calls on unrelated
objects cannot match, and litellm.utils. accepted since four call sites reach
get_secret that way.
Widening the patterns alone would demand about 320 new rows in the central
reference table, most of them provider credentials that are already documented
on their own provider pages. So the gate now looks across every page of the docs
site rather than only that one table, which leaves 143 keys genuinely
undocumented instead of 322.
