A batch or fine-tuning job is created from a file the caller already uploaded,
and that file only exists under the credentials of the deployment that stored
it. When the router fell back to a different model group it handed that file id
to a provider that has never seen it, so the caller got the second provider's
complaint about the file id instead of the error that explains what was actually
wrong with their request.
run_async_fallback now skips fallback targets outside the original model group
whenever the request carries input_file_id or training_file. Order-based
fallbacks stay inside the group, so retrying across deployments still works.
The same handler also crashed with "'NoneType' object has no attribute 'update'"
whenever a fallback fired on a request with metadata set to None, which
/v1/batches always does when the caller sends no metadata, turning the provider's
400 into a 500. Record the model group with a merge instead of setdefault, and
write it to litellm_metadata on the endpoints that use it so the router's
bookkeeping no longer lands in the metadata stored on the provider's batch.
* fix(proxy): return the real status code when a credential update is rejected
update_credential ended its except clause with 'return handle_exception_on_proxy(e)'. Returning the exception makes it the response body, so FastAPI answers 200 and every rejection on this route reads as a successful write to any caller that checks the status; the admin dashboard's API client is one. Patching a name that does not exist answered 200 with the real 404 buried in the body.
The sibling handlers in this file already raise. The route had no test coverage, which is why it survived.
* Update tests/test_litellm/proxy/credential_endpoints/test_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Guards _declared_query_params against a regression in the get_flat_params
migration: the flatten step returns path, query, header and cookie params
together, so a dropped ParamTypes.query filter would wrongly treat path or
header names as declared query params and accept unknown ones. Removing the
filter fails these tests.
fastapi 0.140.7 removed get_flat_dependant(), which broke the import in
management_v1/common.py and took down every /management/v1 route. Switch to
get_flat_params() and filter to ParamTypes.query so unknown-query-param
rejection keeps matching the old behavior.
The end-to-end extras test now injects a MagicMock(spec=HTTPHandler) via
the client parameter instead of patching post on a real handler, and the
docstrings on the new regression tests are removed, addressing the
remaining Greptile review feedback.
Mirror the chat extras translation for /v1/completions, adapted to the
typed OpenAI SDK: anything completions.create() rejects (reasoning_effort,
response_format, fireworks-native extras) rides inside extra_body, which
the SDK merges server-side. Top-level reasoning_effort and response_format
are moved into extra_body (they raised TypeError before), truncate
aliases, chat_template_kwargs effort keys, and guided_* resolve into
extra_body fields, and the strip set removes the rest. Verified live:
/v1/completions rejects prompt_truncate_len, so both truncate names are
stripped on this path rather than renamed.
The Azure Sentinel logger hardcoded the commercial Entra authority and the
commercial Azure Monitor audience, so Log Analytics ingestion could not work in
Azure Government even when the ingestion endpoint pointed at a sovereign Data
Collection Endpoint.
Resolve the authority from AZURE_AUTHORITY_HOST and derive the matching Logs
Ingestion audience from it. Moving only the token URL is not enough: sovereign
Entra would then be asked for a token scoped to the commercial audience, which
the sovereign endpoint rejects.
The http handler merges extra_body after transform_request, so a
response_format nested in an explicit extra_body would silently clobber
the explicit top-level response_format. Drop the nested copy with a
debug log so the top-level value wins, closing the precedence hole in
the guided-param native-wins path.
Streaming /chat/completions and /v1/responses emit nothing, not even response headers, until the upstream yields its first chunk, so an ingress with an idle read timeout (nginx proxy-read-timeout) drops long time-to-first-token streams.
Reuses the existing Anthropic keepalive wrapper with a configurable ping payload, emitting an SSE comment on the OpenAI-shaped routes so conformant clients ignore it. Off unless litellm_settings.sse_keepalive_ping_interval_seconds is set.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_get_deployment_credentials_with_provider_bedrock_batch_fields already
covers s3_encryption_key_id on the base branch, and the new test passes with
every production file in this branch reverted, so it guards nothing.
create_websocket_passthrough_route existed but /openai and
/openai_passthrough only registered HTTP methods, so WS upgrades were
rejected at routing. Add catch-all websocket routes mirroring the HTTP
passthrough target construction.
Fixes#36088
An intercepted web search called litellm.asearch() with only the search tool's litellm_params, so the search request carried no owner. The proxy's spend hook skips any call with no key, user or team attached, so the search's provider cost never reached SpendLogs; it was missing from the Logs page and never counted against the caller's budget. The same path never ran the rate limiter either, so an intercepted search was free of the key's RPM/TPM limits.
The search now carries the originating key's attribution metadata (key hash, alias, user, team, org, plus model_group set to the resolved search tool) and runs the caller's rate limit checks before hitting the provider, matching what a direct /v1/search request gets. SDK calls with no proxy auth context are unchanged.
Resolves LIT-5033
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The staging merge tightened the LIT002 ceiling, so the three mutable dict
literals this branch added now breach it. Build the S3 request headers as
MappingProxyType and resolve the encryption key from a tuple of sources.
create_a2a_client took the raw client off a process-wide cached handler and
called headers.update() on it, then leaned on folding the header set into the
cache key (through the unrelated disable_aiohttp_transport field) to keep one
caller's credentials away from the next.
Per-caller headers now ride with each request through the a2a SDK's call
context, and the agent card fetch gets them through resolver_http_kwargs, so
the shared client is never written to and its cache key no longer varies by
header set. Since the proxy puts a fresh trace id in every request's headers,
that key previously changed on every call, giving each request its own httpx
client and flushing the 200-entry client cache that every other provider
shares. All A2A callers on one timeout now reuse a single pooled client.
Sharing that client also means sharing its httpx cookie jar, which httpx fills
from every Set-Cookie and replays on any later request to a matching domain, so
one agent's session cookie would arrive at another agent on the same host. The
pooled client now carries a cookie policy that stores and sends nothing, which
neither litellm nor the a2a SDK relies on: the SDK's auth interceptor skips
cookie-borne API keys outright.
Closes GHSA-6hr6-w5qg-qmwg (CVSS 5.3), the second finding from the same
osv-scan run as the js-yaml bump. Bundled here so the scan goes green in
one merge instead of two PRs that each stay red on the other's finding.
Re-derived with `uv lock --upgrade-package h2` rather than taking the
Dependabot lock wholesale: that keeps the diff to the two packages that
actually move (h2, plus hpack 4.2.0 which h2 4.4.1 requires) and leaves
the `exclude-newer` snapshot a real timestamp.
h2 4.4.1 published 2026-08-03, hpack 4.2.0 on 2026-06-23 — both clear of
the 3-day exclude-newer window.
The constraint was 1.0, so anything above that was silently clamped down.
SCX accepts [0.0, 2.0), verified live against both GLM-5.2 and Qwen3.8
Max: 1.5, 1.99 and 1.999 all return 200, while 2.0 returns 400 with
"Temperature should be in [0.0, 2.0)"
Since the clamp is an inclusive min(), 2.0 cannot be the ceiling or it
would pass through a value the endpoint rejects. 1.99 is the practical
maximum
The clamp test now pins both ends: 2.5 comes back as 1.99, and 1.7 rides
through untouched where it used to be flattened to 1.0
Replaces the five launch models with the two that SCX.ai now leads on.
Both are live on api.scx.ai and both were verified against it for tool
calling, json_object and json_schema output, reasoning, prompt caching,
and, for Qwen3.8 Max, image input
Pricing follows SCX's published USD rates. GLM-5.2 lands at $0.55/M input
and $1.9255/M output, tracking the recent GLM-5.2 market repricing;
Qwen3.8 Max at $1.815/M and $5.4461/M sits under the only other seller of
that model, and is the first Qwen3.8 Max entry in the catalog
Also corrects a metadata bug the removed entries carried: they set
max_tokens equal to max_input_tokens, conflating the context window with
the output cap. Both new entries declare a max_output_tokens of 131072,
which is what the endpoint's own validator enforces
The Add Model placeholder moves to scx-ai/GLM-5.2 now that MiniMax-M2.7
is no longer in the catalog
Closes GHSA-5p4m-2wfm-xmqj (CVSS 7.5), flagged by osv-scan against
ui/litellm-dashboard/package-lock.json. js-yaml is pinned by an exact
npm override, so the override and the lock move together.
Dev-only dependency: js-yaml reaches the tree through eslintrc, knip
and @redocly/openapi-core, none of which ship in the built dashboard.
4.3.1 published 2026-07-31, clear of the 3-day min-release-age cooldown.
* fix(proxy): re-assert the authenticated identity on passthrough requests
The passthrough merges the client's litellm_metadata into the request metadata
and then re-asserts only user_api_key and the parent span. Every other identity
field the spend and budget pipeline reads stays whatever the request body set,
so a body carrying user_api_key_user_id, user_api_key_team_id,
user_api_key_org_id or user_api_key_end_user_id charges that user, team, org or
end user instead of the caller.
Re-assert the whole sanitized identity after the merge, so the client's copy of
any of those fields is overwritten by the authenticated key's own values.
* test(passthrough): assert no authenticated identity field is client settable
The existing regression names seven fields; the re-assertion covers every field
get_sanitized_user_information_from_key returns, which is twenty today. Derive
the set from the helper so a field added to StandardLoggingUserAPIKeyMetadata is
covered without touching the test.
Two of the twenty were not covered before, including user_api_key_hash, which is
distinct from user_api_key and was client settable.
