mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-01 07:03:02 +00:00
55611 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
a247d9e553 | test(guardrails): run the failure-log time-bound tests on controlled clocks | ||
|
mateo-berri
|
d9b1bafc17 |
Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts: # litellm/litellm_core_utils/litellm_logging.py # tests/unit/litellm_core_utils/test_streaming_handler.py |
||
|
mateo-berri
|
4042a605af | fix(logging): keep the fallback's deployment in success and spend logs when a rejected attempt's failure log is masked | ||
|
tin-berri
|
17d1f58336
|
fix(ui): preserve team model grants on unrelated settings saves (#49258) | ||
|
Etienne Chabert
|
372c096388
|
fix(proxy): evict a deleted key from every worker's auth cache (#47609)
* fix(proxy): evict a deleted key from every worker's auth cache /key/delete dropped the key object only from the in-memory cache of the worker that served the call. Auth serves key objects cache-first, so every other worker, and the Redis copy when enable_redis_auth_cache is on, kept authenticating the deleted key until user_api_key_cache_ttl expired. /key/update already evicts and broadcasts through evict_and_broadcast, delete now does the same for the tokens it deletes * fix(proxy): evict only the deleted rows and broadcast them in one batch Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker * refactor(proxy): drop the comment above the delete eviction call * fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait * test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests * test(proxy): wait for the peer to apply the broadcast instead of polling its cache * test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer * fix(key_management): evict every requested key on delete, including keys whose row is already gone A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone * test(key_management): split the gone-row eviction assertion into one comprehension per worker --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ab3f6caddc
|
fix(gemini): flag computer use support on the four models the Gemini computer use docs list (#49252)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tavren27
|
9432bf6b8e
|
perf(streaming): stop raising caught AttributeErrors inside pydantic on every streamed chunk (#49166)
* perf(streaming): stop raising AttributeErrors inside pydantic on every streamed chunk Every later plain chunk on openai and azure streams raised and caught 12 AttributeErrors inside pydantic. Assigning an undeclared extra goes through a class lookup that raises in the model metaclass and is never memoized, and the hidden_params getter went through the private attribute fallback set_extra_attribute writes the extra and fields_set directly when a cached per-class check says pydantic would do the same, and falls back to setattr otherwise. The hidden_params getter reads the private storage directly. Emitted chunk JSON is unchanged * fix(streaming): match pydantic's own extra-field bookkeeping on every supported version Take the direct extra write only when the installed pydantic itself adds an assigned extra to model_fields_set (2.13 and later) and the class defines no attribute of that name, so pydantic 2.11, 2.12 and 2.14 see the same model state as a plain setattr. A deleted _hidden_params raises AttributeError again, and the mutable test models carry frozen-ok reasons. * fix(streaming): keep the normal write for models that do not allow extra fields openai's construct gives every model an extras dict, so a forbid model built that way still needs setattr's ValueError instead of a silent extra write. * test(streaming): cover a ClassVar declared without a value in the extra-field parity test * fix(streaming): type the streamed choice's logprobs as the parsed value main now stores --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8ac9b75737
|
fix(model_prices): rolling registry audit, add Gemini batch and Azure Astra flex tiers (#49114)
* fix(model_prices): add EmpirioLabs model rows from the provider catalog Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): align EmpirioLabs rows with the registry endpoint vocabulary and schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): absorb Gemini long-prompt batch rates and Azure GPT-6 Astra long-context flex rates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): drop EmpirioLabs rows from the registry audit --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tessellark
|
a084d13635
|
fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures (#49075)
* fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures Pass-through failures on Gemini and Vertex routes carry the model only in the URL, so the failure spend row and the Prometheus failure counter logged an empty model. The failure payload now falls back to the model parsed from the URL on those routes, and the Prometheus provider label honors the provider the request declares before inferring one from the model name * fix(passthrough): keep model and provider on mid-stream and Vertex failure logs Build the failure payload from the URL at both streamed relay sites so a 200 stream that dies mid-body still logs the model, derive vertex_ai for Vertex routes that never declare a provider, and have the Vertex logging handler reuse get_vertex_model_id_from_url instead of its own regex * fix(passthrough): bound the Prometheus provider label and name Vertex failures like the success handler Declared providers only reach the api_provider label when they name a registered provider, otherwise the label falls back to inference. Vertex failure rows now use the success handler's model naming (vertex_ai/ prefix on rawPredict and streamRawPredict, vertex_ai/search_api on :search) and its host-based provider detection, and the detected-stream failure sites are covered by tests. * fix(passthrough): name Google failures from the URL on untracked methods too A Gemini or Vertex pass-through failure on a method the success handler does not track (countTokens, for one) still logged model "" and no provider, although the model sits in the URL. The failure payload now falls back to the Google AI host for the provider and reads /models/<name> from the URL, so countTokens 400s name their model and provider like generateContent ones do. Non-Google hosts keep the tracked-route rule and still log nothing extra. * fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases The failure path only named the Google provider from the URL when the method was one the success handler tracks or the host was a Google API host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE (or a Vertex one behind a custom base) still logged an empty model. A route that declares gemini or vertex_ai now wins outright, and a URL whose path carries /projects/<p>/locations/<l>/ counts as Vertex on any host, while a tracked method on a non-Google host no longer passes as Vertex. * fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model * test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback * fix(passthrough): scope declared provider to pass-through failures and match Google hosts only Prometheus trusts the resolved provider as before and reads the request-declared provider only on pass-through failures. Vertex attribution matches Google hosts, strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the URL path only, covers Vertex Live WebSocket failures, and labels a router alias failure with the alias the client requested * fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler. * fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures A pre-call guardrail block on a pass-through route now builds its failure payload the same way upstream failures do, so the model and provider read from the URL reach spend logs and metrics. Vertex Live websocket failures (connect errors, upstream rejections, and the credential failure) now carry the model from the ?model= query parameter under the vertex_ai provider * fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls Prometheus reads a failure's declared provider only on pass-through calls, so without call_type it fell back to guessing from the model name. That labeled a guardrail block on the Gemini route as vertex_ai and the Live credential failure as None * fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row * fix(pass-through): keep URL model attribution off routes that name another provider A built-in route that names a non-Google provider, like /openai pointed at Gemini's OpenAI-compatible host, no longer takes its failure model from the Google URL. A guardrail block response now returns the x-litellm-call-id of its spend row. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b195d15b93
|
fix(vertex_ai,azure_ai)!: auto-add the compaction, thinking display, and tool change betas on Claude routes (#44747)
* fix(vertex_ai): forward and auto-add the compact-2026-09-04 beta on Claude routes * test(vertex_ai): prove the compact-2026-09-04 beta reaches the wire through both Claude handlers * fix(vertex_ai): move a client-sent body compact-2026-09-04 beta to the request header * refactor(vertex_ai): build the Claude beta set without mutation and inject the client in the messages wire test * fix(vertex_ai): keep client body betas in the body and answer 400 on a malformed anthropic_beta * fix(rust): allow the compact-2026-09-04 beta on vertex_ai in the Rust beta table The beta header config now maps compact-2026-09-04 for vertex_ai, so the Rust table that mirrors it must stop rejecting that beta there, and the Rust Vertex messages route adds it for a compaction request like the Python route does. * fix(vertex_ai): keep client body betas out of the Vertex body next to litellm betas, honor drop_params, and send the compaction beta on count_tokens Litellm's own betas replace a client's body anthropic_beta again, as on main, so a body beta Vertex rejects no longer turns a 200 into a 400. A malformed beta drops under drop_params and names its field otherwise. inline-tools-2026-09-15 joins compact-2026-09-04 as header-only, since Vertex rejects both in the body. Vertex count_tokens sends compact-2026-09-04 when the request carries compaction. * fix(logging): log and bill Anthropic messages compaction responses on the Rust route The Rust route hands the served Anthropic body to the logger, which validates it against AnthropicResponse. That model had no compaction content block and no compaction stop reason, so a compaction reply raised and wrote no spend row. * fix(vertex_ai,azure_ai): forward compact-2026-09-04 to Azure AI Foundry, merge Vertex body betas, and redact compaction summaries Azure AI Foundry needs the compact-2026-09-04 header for compaction, a replayed compaction block, and count tokens, so the beta map, the native compaction providers, and the Rust beta table now allow it there. Vertex keeps a client's body betas next to the ones litellm adds, as the Anthropic route does, and turn_off_message_logging now redacts compaction summaries too * fix(vertex_ai,azure_ai): auto-add thinking display and tool change betas, redact streamed compaction summaries Folds the header changes from #44103 into this PR. Vertex AI and Azure AI Claude routes now add thinking-display-updates-2026-08-18 when thinking.display is "updates" and mid-conversation-tool-changes-2026-07-01 when a tool_addition block is present, matching the Anthropic and Bedrock routes. With turn_off_message_logging on, compaction summaries from streamed messages and streamed chat completions are now redacted in spend logs and callback payloads, like the non-streamed response already was. * fix(vertex_ai,azure_ai,rust): add tool change betas on count tokens and Rust messages, bill Rust server tool replies Vertex and Azure count tokens now add mid-conversation-tool-changes-2026-07-01 when the history carries a tool_addition or tool_removal block, through one shared helper with the compaction beta. The Rust /v1/messages route adds thinking-display-updates-2026-08-18 and mid-conversation-tool-changes-2026-07-01 when a request uses them, on the providers that accept each beta. Rust-route logging parses the served body with the Anthropic response parser, so web_search, pause_turn, mcp_tool_use and container_upload replies get a spend row. Legacy enabled thinking keeps display updates on anthropic too, and the Vertex transform reads drop_params through drops_unsupported_params. * style(rust): rewrap beta map test line with cargo fmt * test(rust): expect the display beta on Databricks, which main's beta list now accepts --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
alvcol
|
d18efdfe13
|
fix(vllm): send the pass-through key as a bearer on both /vllm routes (#46861)
* fix(vllm): send the deployment api_key on /vllm passthrough calls
VLLMModelInfo.get_api_key always returned None, so /vllm/{endpoint} calls to a router
model reached the vLLM server with no auth header even when the deployment had an
api_key, and a server started with --api-key answered 401 (#23443).
Return the deployment key (or VLLM_API_KEY) and send it as Authorization: Bearer, the
header vllm serve --api-key checks. x-api-key was never actually sent before, since the
key was always None.
* fix(vllm): send the pass-through key as a bearer on both /vllm routes
vllm serve --api-key only reads Authorization: Bearer, so a keyed vLLM
answered 401 on both pass-through routes. Send the key as a bearer and
keep x-api-key for gateways that read it. Router models send their own
deployment key and never fall back to VLLM_API_KEY, which belongs to
the /vllm route's own server.
* fix(vllm): send only the bearer on router models and stop doubling /v1 on /vllm
Router models now send the same Authorization: Bearer header /v1/chat/completions sends for that deployment, without the x-api-key copy that route never sent. The VLLM_API_BASE route keeps x-api-key next to the bearer, as it sent it before. A router model whose api_base already ends in /v1 no longer gets /v1/v1 on /vllm/v1 paths.
* fix(vllm): send no placeholder key from keyless router models and stop doubling /v1 on the VLLM_API_BASE route
* fix(vllm): send VLLM_API_KEY when a router model with no api_base is forwarded to VLLM_API_BASE
* fix(vllm): keep a router model's own key on the VLLM_API_BASE server and resolve an oidc VLLM_API_KEY
* fix(vllm): treat an empty key as no key so pass-through sends no empty bearer
* fix(vllm): send no bearer for a Google oidc key and match VLLM_API_BASE across a trailing slash
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
tessellark
|
5d5e3685c1
|
fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933)
* fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths
* fix(logging): keep guardrail and token-count reads of a request from logging drop warnings
Only the real Anthropic-messages adapter request reports a dropped thinking budget. Guardrail,
shadow-eval and token-count translations of the same request now drop it silently, so one request
logs one warning. The 422 retry also no longer hashes each loc entry, so a list inside loc cannot
raise TypeError.
* fix(bedrock): keep the DeepSeek R1 Converse reasoning_effort drop at debug level
Only allowed_openai_params reaches that branch, and it puts reasoning_effort back after the provider mapping, so a warning there claimed a drop that never happened
* fix(logging): warn on Gemini non-integer budget drops and keep the auto-router baseline silent
Gemini 2.x now logs the drop of a non-integer thinking budget. The auto-router
savings baseline builds a body it never sends, so it no longer warns. Logged
values are capped with reprlib, Azure AI marks only its drop_params retry as
gated, and Bedrock Chat Completions names the model without its route prefix.
* fix(logging): warn on the Gemini 3, Magistral, Ollama, Fireworks and DeepSeek ARN reasoning drops
Gemini 3 and later now log a dropped thinking budget, Magistral logs the
level it swaps for a reasoning prompt, Ollama logs a level it maps to
think off, Fireworks logs chat_template_kwargs effort keys dropped for a
model the map does not list as reasoning, and DeepSeek on Bedrock logs a
drop when the model is addressed by an ARN that advertises the param.
* fix(logging): warn on the remaining Ollama, DeepSeek and Tencent reasoning drops
The ollama/ completion route now shares the chat route's think mapping, so
a level it turns off logs the same warning. DeepSeek warns when it drops
thinking.budget_tokens, and Tencent warns when an adaptive-only model gets a
non-integer budget. Magistral no longer warns on low, medium, high or
enabled thinking, the values its reasoning prompt stands in for. The Gemini 3
budget warning no longer claims the model uses its default level.
* fix(reasoning): warn when Claude drops a non-string reasoning_effort
Claude chat and native /v1/messages dropped a reasoning_effort like 5 or
{"effort": 5} with no log line. Both now log one warn-only WARNING. A
Magistral thinking.type sent as a list no longer raises TypeError.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
quietjules
|
df47ec0171
|
fix(spend): keep a NaN or infinite cost from wedging daily spend writes (#48829)
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row. * fix(spend): price non-finite provider costs on every route and drain parked NaN batches Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON * fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert * fix(spend): keep non-finite costs out of headers, responses, and admin JSON Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and streamed frames, so admin writes and logging integrations keep their output. The cost calculator records an overflowing total as 0 with a warning, cost headers leave out non-finite values, a non-streaming NaN usage.cost is dropped instead of failing the response, and a parked NaN spend log row reads back as 0. * fix(spend): keep caller-set call ids from splitting the non-finite cost warning The overflow warning strips newlines from litellm_call_id, and neither non-finite cost warning prints the raw value, which said nothing beyond "non-finite" * fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request * fix(types): drop a usage cost breakdown that holds a non-finite number * fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost * fix(spend): keep the dropped-cost flag out of reach of provider usage keys A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
997d27d27c |
fix(logging): keep the masked failure log body across retries and bound the sync mask
Keep the masked prompt in the spend log body when a router retry or fallback also fails, since every attempt resets the logging object's messages to the raw prompt. Run the sync failure mask on its own thread pool and cancel it when it is still queued at its time bound. A guardrail with only an async logging hook now redacts the sync failure log instead of passing the raw prompt through. Extend the mid-stream failure log wait by the mask time only when a masking guardrail runs, so deployments without one keep the one second wait. |
||
|
tavren27
|
43c07a0f12
|
fix(router)!: check and count deployment itpm and otpm on every Responses WebSocket turn (#49069)
* fix(router): check and count deployment itpm and otpm on every Responses WebSocket turn The success hook only reconciled an ITPM/OTPM reservation found in top-level metadata or the standard logging object, but on the proxy the reservation lives in litellm_params.metadata, so no reservation was ever settled there. The hook now always calls the reconcile, which already returns early without a reservation and clears what it settled, so a second success event is a no-op. A Responses WebSocket turn after the first now has its deployment's current-minute ITPM and OTPM counters checked and is refused with a 429 frame once a counter is at its limit. An admitted turn carries a zero reservation on the current minute's keys, so its actual tokens land on those counters on both the native and the managed relay. * fix(responses): settle the socket open's itpm/otpm reservation on the first turn only After the first turn takes its logging copy, both websocket relays drop the open's ITPM/OTPM reservation keys from the connection metadata, the same way the budget reservation is detached. Later turns, unsolicited backend events, turns after limits are removed mid-socket, and sockets with no turn admission no longer settle or refund the open's reservation a second time. * refactor(responses): read the itpm/otpm reservation keys from core metadata helpers instead of router_utils * fix(responses): keep a socket open's hold for its first real turn and check tokens before request limits A turn that will settle the socket open's itpm/otpm reservation no longer counts that reservation against itself or swaps it for a zero one, so Codex's warmup frame on a managed provider, or a refused first frame on the native socket, no longer blocks the first real prompt or strands the hold for the minute. The token check now runs before the rpm/rpd checks, so a turn refused for tokens never uses up a request slot. Token-check tests pin the clock, and the managed relay settle test drives a real streamed call through a custom provider. * test(router): pin the router clock in the socket turn admission tests Both tests seeded this minute and the next to dodge a minute boundary. They now use the file's pinned-minute fixture and seed only that minute, so neither depends on when the test runs. * fix(router): check a responses socket's open without reserving and read shared token counters first A Responses WebSocket open now runs the same itpm/otpm check a turn runs and reserves nothing. It used to reserve an estimate that falls back to the model's max output, so a socket could not open under any otpm below that. The open still records a zero reservation for its own minute, so the first frame, which the relay forwards without a turn check, counts its actual tokens like every later turn. This drops the plumbing that only carried the open's hold into the first turn. The turn admission callable takes no arguments again, and the native relay detaches only the budget reservation, as before this change. The itpm/otpm turn check reads Redis first and falls back to this instance's in-memory count only without Redis or when the circuit breaker is open. These counters go down when a call settles, so an in-memory copy at the limit could refuse turns after the shared count dropped. The tpm check keeps its in-memory shortcut, since its counter only grows within a minute. Responses usage that reports input_tokens_details.cached_tokens now leaves cached input out of itpm, as chat usage already did. * fix(router): leave a socket turn's cached input out of itpm The native Responses WebSocket relay logs a turn with its list of events, so the deployment rate limit hook finds no usage on the response and falls back to the standard logging payload. That fallback read cached input from metadata.cache_read_input_tokens, a key the payload's cleaned metadata never carries, so every cached token of a socket turn was counted against itpm. The fallback now reads the cached count from the payload's usage_object, the same usage the logging path already built for cost, so a native socket turn counts only its uncached input. The managed relay and HTTP /v1/responses already counted correctly through the response's own usage * fix(router): mark only responses socket opens as check-only and count a turn on the minute it finishes The check-only branch for a socket open keyed on a websocket value in the request kwargs, which any HTTP body can carry and which the realtime route also sets. The router now passes an opens_responses_socket flag into set_io_token_rate_limit_request_kwargs only for its _aresponses_websocket call, and the flag lives in the same context value as the request kwargs, so no request body can set it and realtime opens reserve and get refused as before A zero reservation holds nothing to refund, so a socket turn now counts its real tokens on the minute the call finishes, read from the clock at settle time. A non-zero reservation still settles on the keys it reserved on so its refund hits the same counter When the shared Redis read returns nothing, the check now falls back to this instance's in-memory count instead of admitting the turn with no count at all * fix(router): keep io token reservation keys out of the metadata sent to providers The itpm/otpm pre-call check wrote its reservation keys into both metadata and litellm_metadata. On the routes the router serves through its generic helper (Responses, Anthropic Messages, and the rest), metadata is the client's own field and goes to the provider as is. So _litellm_itpm_cache_key and _litellm_otpm_cache_key, which carry the deployment id and the model name, reached the provider, came back in the echoed response metadata, and counted toward the provider's 16-key cap. The managed Responses socket relay merged each turn's reservation into the frame's metadata the same way, and a native socket opened with metadata sent the keys on every turn through its request defaults The router now passes the limiter the metadata field it uses for its own fields on that call (metadata for chat, litellm_metadata for the generic routes), and the reservation goes only there. metadata_channel is a required keyword, so a new caller has to name the field it writes. The managed relay puts each turn's reservation only on litellm_metadata With the reservation only on litellm_metadata, a retried call on the generic routes refunded its first attempt twice and pushed the deployment's counters below zero. The failure hook read the reservation from the logging copy in litellm_params.metadata and cleared only that copy, so the router's litellm_metadata still held it and the retry refunded it again. The hooks now also read and clear litellm_params.litellm_metadata, which is the router's own dict The proxy does not strip client keys that start with _litellm_ on the way in, so dropping that prefix at the provider boundary would also drop keys clients send today The io token tests now run Redis in process with fakeredis instead of a TCP fake server * test(responses): count socket turns through real logging callbacks instead of a Logging subclass The native relay test now hands the rate limit check to a real Logging object as its success and failure callback, so the turn's tokens go through the same dispatch path production uses. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
4786dd861b
|
fix(proxy)!: scope /health by wildcard keys and team model allowlists the way auth does (#49155)
* test(proxy): cover /health scoping for wildcard keys and team model allowlists * fix(proxy): scope /health with the model access check auth runs on a request * fix(proxy): scope /health to the model group an allowed name routes to A key alias, team alias, or router model_group_alias lets auth admit a requested name by checking its target, so /health also kept the deployment under the requested name, one the key's requests never reach. Keep only the served model group each admitted name dispatches to, through the same alias layers /v1/models mirrors, and let alias_map read its own read-only output. * test(proxy): cover /health skipping a deployment that has no model name for a restricted key * fix(health): show deployments reached through wildcard and alias allowlists, and by id when an alias shadows the name * chore(router): drop should_include_deployment from the outside-use allowlist now that /health no longer calls it * fix(health): never list a default-fallback deployment for an unserved grant, and keep access-group keys to their group's deployments * fix(health): resolve ?model= through the caller's aliases the way chat does * fix(proxy): apply user and agent model limits to /health and yield between checks /health now runs the same user and agent model checks chat runs, so a personal key whose owner may call only some models, or an agent key limited by its access group, no longer sees or probes other deployments. Each per-name check yields to the event loop, and ?model= reaches the wildcard deployment serving an allowed name * fix(proxy): check owner and agent grants for wildcard routes in /health and fail on non-access auth errors * fix(proxy): answer /health auth lookup failures the way chat does and record ?model= checks under the requested name * test(proxy): pass recorded_model_name in the direct health save tests * fix(proxy): count alias names as /health candidates so aliased names a key may call get probed * fix(health): narrow wildcard-served deployments to the key's access groups * fix(health): narrow every deployment serving a name to the key's access groups, team-owned ones included * fix(health): narrow a team deployment's internal name to the key's access groups like its public name * fix(health): keep a team deployment's internal name narrow when a limit holds only that name An agent or key limit that holds one team deployment's internal name no longer reaches every deployment behind the team's public name. The internal name maps to its public name only when the public name is also admitted. * test(health): cover all-proxy-models next to * for a key that also holds an access group * refactor(auth): keep the _as_proxy_exception name so main's tests still find it * fix(health): keep an internal team name a wildcard grant admits as its own target --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
160a0b72cf
|
fix(mcp): bill both calls of a streamed chat completions MCP auto-execute (#48899)
* fix(mcp): bill both rounds of a streamed chat completions MCP auto-execute A streamed /v1/chat/completions request with an MCP tool set to require_approval "never" runs two model calls on one logging object, and the final-stream dedupe kept only the first round's spend row. The final usage chunk also showed only the second round. The MCP stream iterator now holds the first round's assembled response, prices it, and logs one success row that carries the usage and cost of both rounds, with the fixed margin charged once. The final usage chunk shows the same summed usage. A single-round stream still logs its one row when the answer finishes. When the follow-up round fails, the first round is logged from a copy of the logging object taken before the follow-up opened, and the failure row carries only the follow-up's partial usage with no second fixed margin. With proxy post-call guardrails armed, only (assembled, cache_hit) tuples are parked for the proxy, and the deferred log is armed only while the iterator reads, so the proxy's orphaned-callback cleanup never sees it after the Router's first-chunk read * fix(mcp): send the usage chunk of a streamed chat MCP request that ends in one round When a streamed chat completions MCP request ends in its first round (the model answers without a tool call, or no tool results come back), the iterator drained the inner stream to fire its spend log and dropped the usage chunk that stream produced, so a client that asked for include_usage got no usage chunk at all. The drained usage chunk is now held and sent to the client after the final content chunk and before [DONE], the same place a plain stream sends it, so the last usage chunk matches the billed row. A two-round request still drops the first round's usage chunk and sends the merged one from the follow-up, and a stream without include_usage still sends none * fix(mcp): hold round one's log for the guardrail, bill the chunk cost once, and keep round one's id When the follow-up call of a streamed chat MCP auto-execute fails, round one's success log now goes through the proxy's deferred guardrail release instead of straight to the handlers, so a post-call guardrail runs on it and the row carries it. Without a proxy release it logs directly as before The last usage chunk now takes its cost from the combined row's billed total, so a fixed cost margin is counted once in the chunk, as it is on the row. The follow-up's usage chunk is held until the follow-up ends to make that possible The combined row now keeps round one's id, the stream's first chunk id, which matches how a single-round stream is logged * fix(mcp): bill every chat MCP stream round once on failure, hang-up, and sync drop When the follow-up call fails, round 1's success row is now logged on a copy of the logging object that does not share the request's guardrail results, so a post-call guardrail that reports on the failed request counts once instead of on both rows A client hang-up during the follow-up now bills round 1 plus the follow-up's partial answer on one success row with round 1's id and the fixed margin counted once, the same way a plain stream's hang-up bills its partial answer The last usage chunk now carries round 1's usage and cost when only round 1 is billed, and a synchronous reader that drops the stream mid-answer bills round 1 on the loop it read with instead of discarding it The deferred release's logged_on argument is typed and every release closure accepts it, and the parked dispatch constants and protocols both MCP billing paths use now live in one module. The unit tests wait on a logging signal instead of a fixed delay and check the billed row through the proxy stream hook instead of private callback state * test(mcp): drive the unassembled follow-up through an upstream audio delta that is not base64 The test replaced litellm.stream_chunk_builder with a stub that raises. Round 2 now streams an audio delta whose data is not valid base64, so the real builder raises, the stream parks only best-effort usage, and the client's last usage chunk must carry round 1's tokens and cost * fix(mcp): count the fixed margin once in the combined stream row's usage cost * fix(mcp): bill round 2's partial answer when a client hangs up on a routed stream On the proxy the Router wraps the MCP stream, and when a client hangs up during round 2 the router's own cleanup closes that stream before the proxy's disconnect cleanup runs. The close logged round 1 alone, so round 2's partial answer was never billed. The MCP stream now bills round 1 plus round 2's partial answer as soon as the hang-up's cancellation reaches its read, before anything closes it * fix(mcp): count the follow-up's own messages in round 2's partial prompt The follow-up stream shares the request's logger, so it took the client's original messages as its prompt. When round 2 ended early, from a hang-up or a broken upstream, its partial usage counted only those messages and left out the assistant tool call and the tool results the follow-up actually sent. The follow-up stream now counts the messages the follow-up call sent * fix(mcp): bill round 2's partial when a routed stream's client hangs up between reads On a proxy the stream the disconnect cleanup holds is the router's fallback wrapper, so the cleanup never found the MCP stream's in-flight billing and released round 1 alone when the client hung up while the proxy sat at a yield. The round billing now registers its in-flight biller on the request's logging object, next to the parked release, and the cleanup looks it up there, so any wrapper chain bills round 1 plus round 2's partial once * fix(mcp): send the MCP stream usage chunk under the billed row id and count the partial off the loop A streamed chat completion that runs a LiteLLM-hosted MCP tool bills both model calls on one row under round 1's chatcmpl id, but the usage chunk it sent the client was round 2's held chunk and kept round 2's id, so a caller could not join the usage chunk to its row. The usage chunk now carries round 1's id in every path that sends it, with or without a deferred post-call guardrail, while its usage and cost and every content chunk's id stay as they were When the client hangs up during round 2, assembling round 2's partial answer counts the follow-up messages, tool result included, with the local tokenizer. That ran on the event loop and stalled every other request on the worker for about a second on a large tool result. The assembly now runs on a worker thread through asyncify from both the reader's cancellation and the proxy's disconnect cleanup, shielded from cancellation, and round 1 is checked again after the await so a release that happened meanwhile is never billed twice * test(mcp): wait on the loop with a hang guard instead of a 300 ms deadline * fix(mcp): give a proxy-rejected follow-up's failure row round 1's own cost breakdown * refactor(streaming): fold the partial response's empty guards into one check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
cbfb1abdb3
|
fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555)
* fix(proxy): name shared-handler errors by the status they answer
handle_exception_on_proxy pinned type=internal_server_error on every
exception it wrapped, so a management route rejecting a caller's own
argument answered {"type":"internal_server_error","code":"400"} and an
SDK branching on the type read its own bad request as a gateway outage.
Derive the type from the status the response actually carries, keeping
a type the exception already names.
* test(proxy): pin a 500 for exceptions whose status is None
* fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors
Provider 5xx and timeouts keep internal_server_error so retry loops still retry them,
the shared handler redacts internal hosts from raw exception messages, and /key/list
routes its HTTPException through the shared handler instead of its own copy of the
old label.
* fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures
* fix(proxy): pass shared-handler error messages through as on main
Redacting timeout and 5xx messages changed what every shared-handler route
returns, which relabeling the error type does not need, so messages go back
to the verbatim text main sends.
* fix(proxy): redact only the status-less failure main never sent
* fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403
A failure with no status now answers "Internal server error" instead of its own text, since that text can
name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The
/user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises
* fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise
A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces
* refactor(proxy): parse a three-digit string status without recursing
|
||
|
Siddhant Badola
|
b71491c6c9
|
fix(bedrock): route a Bedrock model id with no Invoke provider to Converse (#43290)
* fix(bedrock): route unmapped converse-only family model ids to converse #42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from falling to the invoke route. The same gap exists for the other families that have no invoke transformation at all: minimax, nvidia, writer, zai, google and xai. An id from one of them that is newer than the cost map hits get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None -> BedrockError, when Converse would have served it Add a second family regex for those six and default them to converse, the same way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because the prompt-caching rule is openai-specific Every shipped id of these families already resolves to a converse entry via its base model (checked against the cost map), so this only changes behaviour for ids that are not in the map yet Tests: six unmapped ids route to converse, and an unmapped meta.* id still routes to invoke since meta has an invoke handler * fix(bedrock): route a model id with no Invoke provider to Converse A bedrock/<model> id missing from the cost map fell to the Invoke route, where an id that names no Invoke provider (a new vendor's model, a bare provisioned-model, imported-model, or custom-model-deployment ARN) fails locally with "Unknown provider=None" before any request is sent. Such ids now go to Converse, which replaces the per-family regex with the general rule. Ids that name an Invoke provider and explicit route prefixes are unchanged. * test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests * fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id * fix(bedrock): answer 404 for an invalid model id instead of AWS's 400 AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown * fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list * fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there * fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows The regex check now strips route and region prefixes from the model and its base_model before asking whether LiteLLM knows the model, so bedrock/converse/<id> gets the same answer in a direct transform call as on the completion path. A cost-map row counts only when it names a provider, so a price-only register_model row leaves an unlisted id unknown, as on the proxy. AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping an identical copy. * fix(bedrock): treat a region path like the bare id when the cost map does not list it A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools, reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works on it and a mistyped one still answers 404. * fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local * fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local * fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse. atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do. * fix(text completion): keep /v1/completions error mapping as it is on main Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a44e44b980
|
fix(cache_control): pick the chat-path dialect from the deployment's provider prefix (#45295)
* fix(cache_control): resolve the chat-path dialect with the deployment api_base * test(cache_control): type the chat-path resolver helpers * fix(cache_control): pick the chat-path dialect from the deployment's provider prefix Resolving with the deployment api_base followed the dispatch's azure_ai to azure flip, so chat on a non-Foundry Azure host sent Anthropic cache_control while /v1/messages, told azure_ai, sent the OpenAI breakpoint. Use the model's provider prefix when the caller does not name a provider, so chat matches the told path whatever AZURE_AI_API_BASE holds, and keep name-only resolution everywhere else. * fix(cache_control): keep litellm_proxy routing when the chat path reads the provider prefix With USE_LITELLM_PROXY or litellm.use_litellm_proxy on, every call goes to the upstream proxy, so the provider prefix no longer decides the cache format there and resolution matches main. The plain OpenAI check reads the same provider value as the hosted check, so one function no longer mixes two lookups * fix(cache_control): resolve the chat-path dialect with the deployment api_base, as the call is routed The hosted cache-format check now resolves the provider with the deployment's own api_base when the caller names none, the same provider the call dispatches to and the Responses path passes, so AZURE_AI_API_BASE no longer decides the chat format. The plain OpenAI check keeps main's name-only lookup. * fix(cache_control): send acompletion fallbacks the caller's request, not the primary's hooked kwargs acompletion handed async_completion_with_fallbacks the kwargs the cache seed and prompt hook had already rewritten for the first model, so a Claude fallback behind a Foundry GPT model received prompt_cache_options and got a 400. Fallbacks now get a copy of the caller's model, messages, tools, and kwargs taken before the hooks, as completion already does with its args. Both cache seeds also read base_url before api_base, since completion sends the call to base_url when both are set. * fix(cache_control): keep the caller's tools list intact for acompletion fallbacks The vector-store hook pops recognized file_search tools from the request's tools list in place, so the fallback request built from the pre-hook snapshot lost the retrieved context. Snapshot a copy of the list instead. * fix(cache_control): skip acompletion's prompt hooks when fallbacks will run them Each fallback attempt runs the prompt hooks for its own model, so running them first in the outer call only cost a vector-store search whose result the hand-off threw away. * fix(cache_control): run acompletion's prompt hooks for fallbacks and leave only the cache hook to each attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
87fec5467f
|
fix(sso): keep CLI SSO and Claude Code gateway team lists in the user record's order (#49033)
* fix(sso): bind CLI SSO and Claude Code gateway logins to the first team on the user record * fix(proxy): keep SCIM and JWT team sync in record order, and look up SSO groups once SCIM PATCH and JWT team sync saved a user's teams from a set, so the first team on the record, the one a Claude Code gateway sign-in binds, could change on any sync. Both now keep the record's order and append new teams in the order the IdP sent them. SSO group sync now drops claim groups that are not LiteLLM teams with one lookup before adding the missing teams one at a time, so a claim full of non-team groups no longer costs one failed add per group on every sign-in. * test(sso): expect the gateway team dropdown in the user record's order * fix(sso): read team existence from the writer, keep new-user and SCIM replace team order * fix(scim): keep the teams a user already has in record order on PATCH and PUT * fix(scim): keep record team order when a SCIM POST matches an existing email * fix(scim): join new teams in request order so a failed roster write keeps it * fix(scim): let a groups replace win over later group ops in a PATCH again Applying group adds and removes after a replace made a PATCH that deactivates a user and adds a team that does not exist answer 404 and leave the user active, where it answered 200 before. Keep the record's team order, but ignore group ops after the last replace as the base did. * test(sso): type the CLI team detail row helper --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: rj-cairn <349020452+rj-cairn@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
02ef44c388
|
fix(streaming): convert upstream logprobs so streams with logprobs end with their finish chunk (#33549)
* fix(streaming): normalize provider logprobs to prevent MockValSer serialization crash * fix(streaming): convert upstream chunk logprobs to litellm ChoiceLogprobs so the finish chunk serializes A streaming chat completion with logprobs on the OpenAI SDK path attached the SDK's own ChoiceLogprobs object to the outgoing chunk when the chunk had no content to rebuild from, which is the provider's finish chunk (empty delta, logprobs for the end token, finish_reason). The SDK parses stream chunks without building that nested class, so its pydantic serializer is still a MockValSer and model_dump_json fails. The proxy then sent an error frame in place of the finish chunk, and when the same chunk also carried usage the SDK wrapper itself raised MidStreamFallbackError. The attach site now validates upstream logprobs through a TypeAdapter that dumps any non-litellm pydantic model before validating into litellm's ChoiceLogprobs, and passes litellm instances through unchanged. * fix(streaming): keep upstream logprobs that don't fit ChoiceLogprobs as plain JSON so the stream still finishes * fix(streaming): build ChoiceLogprobs' serializer up front and keep nonconforming logprobs as JSON on chunk rebuilds ChoiceLogprobs inherits the openai SDK's deferred build, so a fresh worker whose first logprobs arrive on the finish chunk still had no serializer. Foreign logprobs models now dump in full, so a missing top_logprobs becomes [] and SDK callers keep attribute access, and chunk rebuilds for content and usage keep logprobs that don't fit as plain JSON instead of raising. * fix(streaming): keep only the logprobs keys upstream sent and type refusal logprobs * fix(types): pass nonconforming non-streaming logprobs through instead of failing the response --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0070ce23b3
|
fix(router)!: apply content_policy_fallbacks when router settings update at runtime (#45428)
* fix(router): apply content_policy_fallbacks when router settings update at runtime * test(router): annotate the fallback regression test's locals as Final * fix(proxy): validate router_settings.content_policy_fallbacks on /config/update like its siblings * test(proxy): annotate the content_policy_fallbacks config update test locals as Final * fix(proxy): keep litellm_settings content policy fallbacks, accept dict targets, skip invalid stored rules A runtime content_policy_fallbacks rule now layers over the litellm_settings rules instead of replacing them, so models the runtime rules do not name keep their file fallbacks. Fallback targets may be deployment dicts as well as model names. A stored rule that fails validation is logged and skipped at sync, so the other router settings still apply. Per-user token discovery now reads every fallback key from key and team router_settings. * fix(proxy): validate content policy fallback rules one by one and keep a config-owned list alone An invalid stored rule is now skipped on its own instead of dropping every runtime rule, a router_settings list from the config file stays the router's only rules across DB syncs, and GET /fallback names the model of a dict target. * fix(proxy): answer an error for a string-valued fallback rule on GET /fallback instead of its characters * fix(proxy): keep config-file content-policy rules as Router init accepted them and log a skipped rule once per change * fix(proxy): log a skipped content-policy rule whenever the saved rules change * fix(proxy): keep general fallbacks when no content-policy rule is saved, name a model-less target's model, and take only model names in key rules * fix(proxy): reject an empty dict target in a key's content-policy rule --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b8283d8b61
|
feat(usage): show subscription-covered providers and account fees separately (#44772)
* feat(usage): show subscription-covered providers and account fees separately * fix(usage): write only the named fee columns and sync the root schema The fee update now writes just the columns the request names, so two admins changing different fields at once both land, and the dashboard edit form sends only what changed. The root schema.prisma carries the new table like the two copies, and a billing period anchored in the calendar's last month ends on December 31, 9999 instead of failing. * fix(usage): register the subscription table span and keep provider logic under llms * fix(subscription_accounts): register the fee routes as management routes and reject an empty account id * fix(usage): run the new tests in CI and close subscription card review gaps Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee * fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card * fix(usage): count a subscription fee only for accounts behind a current deployment Usage listed a fee row for an account no deployment uses while another ChatGPT deployment existed and dropped it when none did. It now lists only accounts behind a current deployment in both cases. Adds a real proxy and Postgres integration test for the fee API and for ChatGPT Responses rows on a deployment's own login, and names the card's multi-component test with the integration suffix. * fix(usage): keep a stored subscription fee listed after its account's deployments are removed The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
c0ded7b282
|
fix(responses)!: send a dict reasoning_effort the same way on every Responses bridge (#49082)
* fix(responses): send a dict reasoning_effort the same way on every Responses bridge Azure and ChatGPT bridges dropped reasoning mode and context unless the dict also carried summary. Every bridge now runs the reasoning step the OpenAI bridge uses, and each Responses config declares the reasoning values its API accepts, so drop_params drops what a backend refuses. Perplexity ignores mode and context, so it answers 400 for them without drop_params. * fix(responses): refuse ignored reasoning fields on xAI and Fireworks, resolve openai/ names for OpenRouter * fix(responses): refuse a reasoning mode Copilot ignores, keep catalog modes, return refusals as values Apply additional_drop_params before the refusal check, keep a null reasoning field instead of refusing it, let a cost-map row's reasoning_modes and reasoning_contexts override a backend's declared values, and declare GitHub Copilot's accepted values from a live probe * fix(responses): read Azure base_model and deployment rows, accept standard mode, refuse only mode and context * fix(responses): send only effort and summary to an Azure deployment litellm cannot identify under drop_params Azure and Azure AI deployment names are user-chosen, so when neither the name, model_info.base_model, nor a cost map row resolves the model, drop_params now drops mode and context instead of forwarding a pro the model may refuse. Names resolve by their last path segment, so a regional base_model like azure/eu/gpt-5.4 follows gpt-5.4's rows * fix(responses): resolve deployment names by model family and keep reasoning_modes per deployment A custom Azure deployment name now resolves to the longest OpenAI catalog prefix of its base_model or name, an unresolved name forwards mode and context, and a deployment's model_info reasoning_modes and reasoning_contexts apply only to that deployment instead of every sibling sharing its model name. litellm_proxy forwards every reasoning field. * fix(responses): find the model family anywhere in an Azure deployment name and keep only effort and summary for unknown gpt-5 or gpt-6 names under drop_params * fix(responses): look up OpenAI model names inside deployment names only on Azure Databricks endpoints like databricks-gpt-5-5-pro spell versions with hyphens and other backends name local models freely, so only Azure and Azure AI search inside the name, and a run that cuts a hyphen-spelled version never matches * fix(responses): judge Azure AI reasoning fields by the request api_base --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
080e643edc
|
fix(proxy): apply new model rows on every worker as soon as they are written (#49034)
* fix(proxy): apply published model rows on every worker before the debounced resync Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path * fix(proxy): load published model rows from the primary under the reconcile lock Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s. * test(proxy): cover a published model row that names no credential * fix(proxy): keep the unknown model name read-through on the replica and outside the lock Only the by-id load that follows a published write reads the primary inside the reconcile lock. A request for a model name a worker does not know reads through the replica routing before taking the lock again, as on main, so unknown names put no load on the primary and do not wait behind a reload. * fix(proxy): keep access group edits from dropping staying deployments and batch published config ids * fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica * fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing A batch of published model ids that held one auto router skipped every row in it, so ordinary rows written at the same moment waited for the full reload. The by-id load now leaves only the auto router rows to that reload. An access group update now reads every deployment from the primary, answers 404 for an unknown group and 400 for an unknown model id or name before it writes anything, then removes the group from the deployments that leave before it adds it to the ones that join, so a failed write leaves the group narrower than asked, never wider. * fix(proxy): write an access group update in one transaction after a fresh read The update now checks model names first, reads every deployment from the primary, and commits every join and leave in one batch right after that read, so a failed write changes nothing and a field another admin set during validation is kept. Other workers hear about the rows only after the commit. * fix(proxy): lock the model rows while an access group update reads and rewrites them * test(proxy): run the model resync tests on the real installer and a real router The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen. * fix(proxy): lock only the rows an access group update touches and answer 404 before 400 An access group update locked every model row, so a lock on any unrelated row stalled it for 30 s and then failed it with a 500, and unrelated model edits on other proxies waited behind it. It now takes a per-group advisory lock, locks only the rows that carry the group or that it selects, and reads them again once locked. The group existence check runs before the model name check again, so a missing group answers 404 as on main. * fix(proxy): never wait on a locked row in an access group update The update now takes its row locks with NOWAIT. When another edit holds one of the group's rows, it rolls back, holding nothing, and tries again for about 10 s before answering 409. Holding rows while waiting deadlocked against team deletes and let a concurrent model edit undo the update's change. The per-group lock also moves to the two-key advisory space, apart from team ids. * fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select * fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
d1758cda91
|
fix(router)!: keep a deployment's own model_id over the request body's (#49045)
* fix(router): keep a deployment's own model_id over the request body's A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one. * fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component * fix(router): keep a SageMaker deployment's component when it pins it through extra_headers * fix(router): read the SageMaker component header as a pin only on SageMaker deployments * fix(sagemaker): send the inference component only as a header, never as a body model_id SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat container body and legacy parameters no longer carry model_id. Nova endpoints accept a fixed body field list, so a pinned Nova deployment could reject the extra field. * fix(health): probe header-pinned SageMaker deployments at their component Health checks bypass the router, so a legacy sagemaker/ deployment pinned only through its inference component header sent no component on /health while routed requests now send it. Also update the two SageMaker integration tests that still asserted the old precedence and the dropped body field. * fix(health): apply the router's model_id pin rule to health probes Health probes now share the router's pin function, so a deployment that lets clients set model_id gets no header pin and its own model_id wins without reading the header. Header keys that are not strings are skipped when reading the inference component header instead of failing the call. * fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
373f1752d3
|
fix(anthropic): send PDF image_url content to Claude as document blocks (#46984)
* fix(anthropic): convert PDF image_url content to documents Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt_templates): strip content-type parameters from fetched media Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(prompt_templates): format MIME normalization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
4c8b90945f
|
fix(realtime)!: send the Gemini Live setup at the first content so session.update settings apply (#39864)
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its default audio setup at connect, so a client's session.update (GA output_modalities or beta modalities) was dropped and every answer came back as audio. The setup now goes out on the client's first message: a session.update becomes that setup, anything else gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a provider hook. Text setups label deltas response.output_text.delta, and downgrading TEXT to AUDIO on an audio-only model logs a warning * fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup * fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup Gemini and Vertex Live accept one setup per connection, so the bridge used to turn only the first session.update into that setup and drop the rest without telling the client. Now every session.update that arrives before the first content frame is held, deep-merged (later values win per field), acknowledged with a session.updated built from the setup it would produce, and the merged session becomes the one setup sent with the first conversation item, audio frame, or tool result. The session events also surface the systemInstruction text as instructions, which the old string check never matched because the setup carries it as a Content object. * fix(realtime): cap the session.update merge depth so the recursive detector accepts it _merge_session_fields recurses once per nested mapping level, so CI's recursive_detector flagged it. Bound the descent at eight levels (the OpenAI session object nests four deep at most), replace anything deeper wholesale with the later session.update's value, and list the helper with the other depth-capped functions the detector allows. * fix(gemini-live): accept a string output modality and drop values Live cannot produce A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once. * fix(gemini realtime): answer an error for session updates Gemini Live cannot apply An output modality the model cannot produce, or a session.update sent after the first content, now gets an invalid_request_error event instead of a silent swap or drop, unless drop_params is set. The synthetic session.created reports the model's default modality, setupComplete no longer adds an extra session.updated, and merging nested session updates is linear in their size * fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns A pre_call guardrail blocking the first typed message now goes out after the setup, so Gemini and Vertex no longer close the session with 1007. Push-to-talk sessions (turn_detection null or create_response false) wrap each audio turn in empty activityStart and activityEnd messages, which Gemini needs when automatic activity detection is off. A session with no content after 60 seconds sends its setup, since Vertex refuses a setup 300 seconds after connect. Held session.update frames past 10 MB answer session_update_too_large, and deferred mode now errors on a second session.update unless drop_params is set. * fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection An update that leaves the applied setup unchanged now gets session.updated instead of session_update_not_applied, so clients that resend their session with the first response keep working. Deferred setup mode merges held updates like the default mode, turn_detection null turns Gemini's activity detection off, drop_params drops a voice outside the documented Live voices, and the lazy setup deadline moves to 240 seconds * fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content * fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name. * fix(realtime): hold first content so a following session.update joins the Gemini setup Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too * fix(realtime): check AI Studio Live voices like Vertex * fix(realtime): forward any voice to AI Studio Live models that accept it * fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup. * fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock The deadline and grace tests waited on real 50 ms timers, and the merge cost check compared wall-clock runs. The session now reads an injected clock, time.monotonic by default, the two tests step it, and the merge cost check moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes the backend through wrap_backend, which main added after the test was written |
||
|
Mateo Wang
|
73aaf9e3da
|
fix(mistral): forward prompt_cache_key to the Mistral API (#47432)
* fix(mistral): forward prompt_cache_key to the Mistral API * fix(mistral): send prompt_cache_key only to Mistral's API and accept reasoning_effort on devstral and code-agent models * fix(mistral): keep the upstream api_base out of the prompt_cache_key rejection message * fix(mistral): honor allowed_openai_params for prompt_cache_key and gate codestral async and env hosts * test(mistral): annotate the new adapter prompt_cache_key tests * fix(mistral): drop or reject a prompt_cache_key the Mistral API cannot take A non-ASCII prompt_cache_key gets a 500 from api.mistral.ai, and a non-string one a 422, so both now drop under drop_params and answer 400 without it, like any value the provider cannot take. The /v1/messages adapter derives no key from a non-ASCII Claude Code session for Mistral, and the 400 names the caller's provider, so a codestral deployment's error says codestral. * fix(mistral): send a reasoning_effort object as its level and keep UnsupportedParamsError on sync calls * fix(mistral): keep async codestral file uploads working by not rewriting the caller's file blocks |
||
|
Filip Sinitsin
|
e81dd348a4
|
fix(proxy)!: log and bill Responses-style bodies sent to /cursor/chat/completions (#30277)
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126) The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests. * test(proxy): pin provider metadata and key billing on /cursor/chat/completions A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land * fix(proxy): pick the Cursor route's metadata field from the body shape A Responses-style body on /cursor/chat/completions keeps proxy metadata in litellm_metadata, the same as /v1/responses, while a Chat Completions body keeps metadata, the same as /v1/chat/completions. Adding the whole route to the internal-metadata list let a client's chat-body tags replace the key's tags in spend logs and skip tag budgets. * fix(proxy): keep get_metadata_variable_name callable with only the request * fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field * fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies * fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string * fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
cfb77ecc3a
|
fix(proxy): list only routable models for a wildcard alias over a provider org (#49152)
* fix(proxy): list only routable models for a wildcard alias over a provider org A wildcard alias over an org-scoped deployment (together_ai/* over together_ai/nosuchorg/*) listed the provider's whole catalog on /v1/models and /model/info, and every listed id routed to a model the provider 404s. A skipped deployment path segment now counts as a routing prefix only when the routed model resolves to a catalog id through the get_model_info lookup order, so responses/, Bedrock converse/, and region paths list as before. Together's model list now also carries the cost map's together_ai rows, skipping the together-ai-* pricing tiers, so an alias over a real org lists the models Together serves today. * fix(proxy): keep flat catalog ids under routes the cost map can't parse, drop retired Together rows * fix(proxy): check skipped path segments only on Together so Hugging Face provider routes keep listing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
56388fd13d
|
fix(count_tokens): count a history that carries a compaction block (#44784)
* fix(count_tokens): count a compaction block instead of answering 500 A message history that replays Anthropic's signed compaction block made /v1/messages/count_tokens answer HTTP 500: neither the Anthropic nor the Vertex AI partner count-tokens request carried the compact-2026-09-04 beta, the provider rejected the block with a 400, the proxy fell back to the local tokenizer, and that raised on the unknown block type. Both provider count paths now add the compaction beta whenever the history carries a compaction block, and the local tokenizer counts the block's summary text like a text block, skipping the signature. * fix(token_counter): skip the tokenizer for empty thinking and compaction text * fix(count_tokens): forward context_management on Anthropic counts and count chat compaction blocks locally Vertex AI compaction support for inference and counting moves to its own PR, so the Vertex count handler is back to main here * fix(anthropic): map list-form context_management to edits for count_tokens and inference betas Count requests forwarded the OpenAI list form of context_management unchanged, while inference maps it to Anthropic edits, and the inference beta helper read the list as no edits. Both now map it once. A replayed compaction block with an empty summary no longer reaches the tokenizer. * fix(anthropic): forward count context_management only with edits, map Vertex list-form betas, and return provider context_management * fix(anthropic): drop an unmappable context_management list under drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Devon Krisman
|
953750ed41
|
fix(websearch-interception)!: stream native server-tool blocks to clients (#40967)
* fix(websearch-interception): stream native server-tool blocks to clients Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code) never received the server_tool_use and web_search_tool_result blocks the interception layer builds for them, so citation panels stayed empty and result counts read zero. Two gaps in the same flow: FakeAnthropicMessagesStreamIterator._create_content_block_chunks only knew text, thinking, redacted_thinking and tool_use; any other block type emitted a bare content_block_stop with no start and no payload, silently dropping the block from the fake stream. server_tool_use now streams as content_block_start plus input_json_delta and any other server-generated block rides whole in content_block_start, matching Anthropic's native streaming of these types. try_short_circuit_search only recognized a native client by finding a web_search_* tool in the tools list, but the pre-request and deployment hooks convert that tool to the LiteLLM standard shape before the short-circuit runs, recording the conversion in WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so the synthetic response carries the native blocks on the converted path instead of only when hook ordering happens to leave the tool untouched. Fixes #35333 * fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500 * fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply * fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply * test(websearch-interception): build the cost test's logging object with a datetime start time * fix(websearch-interception): send search snippets once in the direct search answer * fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta. * fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite A legacy hook that masked a tool call's arguments on a buffered stream made the whole rewrite undeliverable, text included, so the original stream went out. With web search blocks in the stream that now hit any hook masking the search query. The adapter hands same-count tool-call rewrites to the translation's write-back, as a unified guardrail's already are, and still discards a rewrite that adds or drops a tool call. * fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled ModelResponse, which keeps web_search_tool_result blocks only in provider-specific fields, so a hook masking result titles on a stream never saw them. The hook's view now carries those blocks, its rewrite of them goes back through the ended-stream write-back, and the adapter no longer needs to pass texts outside the hook's view through untouched. * test(guardrails): type the legacy hook stream test helpers * fix(guardrails): scan a streamed web search query as text like the non-streaming reply * test(guardrails): type the masking guardrail test helpers * fix(guardrails): keep the block id when scanning a streamed web search query --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4a5c5665fa
|
fix(azure_ai)!: keep OpenAI prompt cache breakpoints on the Foundry chat completions path (#45299)
* fix(azure_ai): keep OpenAI prompt cache breakpoints on the Foundry chat completions path * fix(azure_ai): keep only OpenAI content part types when preserving prompt cache breakpoints * fix(azure_ai): share the breakpoint check with the cache-mark step and keep only text parts A deployment's model_info opt-in is registered under its provider-prefixed cost-map key without a litellm_provider, so that prefix now names the provider the flag speaks for. * refactor(azure_ai): read text parts through a walrus and mark new test values Final * fix(azure_ai): read a router-registered breakpoint opt-in only for azure_ai and keep a message mark off dropped file parts * fix(azure_ai): read the breakpoint opt-in from each deployment's own model_info and base_model A router deployment without its own supports_prompt_cache_breakpoint flag now reads the cost map as loaded, so a sibling's opt-in on the same model string no longer turns its requests into 400s. The shared hook no longer reads router registered keys for azure_ai. * fix(azure_ai): send breakpoints in Foundry's shape under drop_params and fall back to the model name * test(integration): expect the injected breakpoint on the Foundry GPT-6 chat wire --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
dbcacc7ab7
|
fix(health): probe only models a partial wildcard serves (#49060)
Some checks are pending
unit / router (push) Blocked by required conditions
unit / rust-bridge-harness (push) Blocked by required conditions
unit / auth-checks (push) Blocked by required conditions
unit / budgets (push) Blocked by required conditions
unit / custom-logging (push) Blocked by required conditions
unit / db-and-spend (push) Blocked by required conditions
unit / endpoints-and-responses (push) Blocked by required conditions
unit / enterprise-routing (push) Blocked by required conditions
unit / guardrails-hooks (push) Blocked by required conditions
unit / jwt-and-keys (push) Blocked by required conditions
unit / key-generation (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / proxy-server-core (push) Blocked by required conditions
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(health): probe only models a partial wildcard serves A partial wildcard health check now probes the cheapest cost-map chat models the wildcard matches, keeps provider route prefixes such as Bedrock converse/ on the probe, and falls back only within the wildcard. A wildcard with no matching cost-map chat model reports an error naming model_info.health_check_model * fix(health): keep the provider-wide probe for an unmatched wildcard, skip retired models, and ignore Bedrock ARNs A partial wildcard with no served cost-map chat model falls back to the provider's cheapest chat models instead of failing, and fails with a message naming model_info.health_check_model only when the provider has none. The cheapest-model picker skips models past their cost-map deprecation_date, and a Bedrock ARN segment is never read as a route prefix. * fix(health): fall back to the provider's cheapest probe when a wildcard's own models fail A partial wildcard now probes up to three of its own cheapest chat models first and then the provider-wide cheapest ones, all from one cost-map scan, so stale or unpriced rows the wildcard matches cannot turn a healthy deployment red. Retired models are read by their resolved cost-map key, both deprecation checks use the UTC date, Bedrock only treats converse, invoke, and region names as route segments, and equal-price ties break by model id * fix(health): cap wildcard health probes at three and rank retired cost-map rows last A partial wildcard now probes two of its own cheapest cost-map chat models and then one provider-wide model, so a failing deployment makes no more calls than before. Rows past their cost-map deprecation date go last instead of being dropped, since some still serve, and the shared cheapest-model picker returns what it returned before. * fix(health): probe a wildcard's own models first, then the same three models main probes A partial wildcard's health check tries its two cheapest own cost-map models, then the provider's three cheapest models main already probes, so a row main shows healthy stays healthy. The deprecation check uses the server's local date, like the Baseten listing * fix(health): keep main's provider-wide probe order so equal-price ties match main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
f32ac94ad1
|
fix(openai): keep the upstream error body on e.response for OpenAI and Azure errors (#49079)
* fix(openai): keep the upstream error response on mapped exceptions so e.response.text has the wire body * fix: keep the upstream error body on 429 and 5xx errors RateLimitError, InternalServerError, BadGatewayError, and ServiceUnavailableError rebuilt their response and dropped the upstream body. They now keep the body and exactly the upstream headers, and an unread streaming response no longer raises. * fix: keep a gzip-encoded 429 or 5xx error readable The kept body is already decoded, so the rebuilt response drops content-encoding instead of decoding the body a second time and raising DecodingError. * fix: keep the upstream headers and drop credentials on every mapped provider error The copied error response now keeps every upstream header and carries a request with only the method and the URL, so no provider key, query secret, or prompt reaches e.request. Timeout and APIError get the response too, and azure/ chat, streaming, embedding, and image errors attach it * fix: keep the upstream error body on gateway timeouts and re-raised gateway errors * test(azure): cover the 404, 500, 502, and 504 error bodies and the missing-key placeholder * fix: keep no e.response on timeouts and errors litellm raises itself * fix: keep the upstream elapsed time on error responses and the upstream body on azure_text errors * fix: keep the upstream body without credentials on Azure content filter errors and message-matched OpenAI errors --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
e136bed4a7 |
fix(logging): give the failure spend log the request body with the masked prompt
The proxy's failure hooks and spend log read the proxy's own copy of the request body, which the failure-log mask never reached, so a failed call's spend row and the UI Logs page kept the raw prompt. The masked body is now lifted onto that request, and a logged copy counts as the prompt when its messages equal it, since the body keeps its own list. |
||
|
devin-ai-integration[bot]
|
ccc56d27ad
|
fix(azure): add batch prices to the unversioned o3, o4-mini, o3-mini, gpt-4o and gpt-4o-mini rows (#49182)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
paveln83
|
b91fc92b6d
|
fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054)
* fix(router): cap the default OTPM reservation for requests without max_tokens * fix(router): reconcile ITPM and OTPM reservations after every successful call * fix(router): read the default OTPM reservation from the served model and make its fraction configurable The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check. * fix(router): refund a failed attempt's io token reservation in make_call The proxy passes one logging object to every retry, and it runs failure callbacks only for the first attempt, so the last attempt's ITPM and OTPM reservation stayed held until the minute ended. Each attempt that raises now gives its reservation back before the retry or fallback * fix(router): count Anthropic cache writes once in ITPM reconcile Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens. * fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction * fix(router): reserve with the deployment cap only through the param the request's API sends * fix(router): never let a deployment's output cap raise the reservation above the default * test(router): move the reservation fraction env var test to the integration suite The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation. * test(router): retry the reservation fraction scenario when the minute rolls over The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute. * test(router): give the rollover retry its own deployment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
DewellNaa
|
342ce6ae4a
|
feat(zai): support response_format and max_completion_tokens, add glm-5v-turbo pricing (#48335)
* feat(zai): fall back to BIGMODEL/ZHIPUAI api keys and add GLM-5.3 aliases and glm-5v-turbo Resolve the zai api key from ZAI_API_KEY, BIGMODEL_API_KEY or ZHIPUAI_API_KEY so deployments predating the zai rename keep working. Accept response_format and max_completion_tokens as supported openai params. Register zai/GLM-5.3 and zai/GLM-5.3-Flash aliases plus the glm-5v-turbo vision model in both the main and backup cost maps * fix(zai): read only ZAI_API_KEY, dropping the BigModel key fallbacks * fix(zai): map max_completion_tokens to max_tokens, correct glm-5v-turbo pricing, drop redundant aliases * fix(zai): reject json_schema response_format, add glm-5v-turbo cache read price * fix(zai): forward every response_format type so allowed_openai_params keeps working * fix(zai): follow json_schema natively on glm-5.3, prompt other models, drop over-limit max_tokens under drop_params * fix(zai): keep BigModel's json_schema as sent and never drop the caller's max_tokens under drop_params * fix(zai): write the prompted JSON schema as JSON so models answer with valid JSON * test(zai): type the zai provider tests and mark their locals Final --------- Co-authored-by: DewellNaa <210961864+DewellNaa@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
733cfeba93
|
fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced (#48953)
* fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced * test(integration): a reported cost is attributed to output in the gateway parity cases * fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects * fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge * fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost * fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill * fix(cost): read the stored breakdown total through the logging object's typed field * refactor(cost): name the providers billed from their own usage fields for what they are * fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
369f3cb5f0
|
feat(cost-map): add baseten joey-1 decision model (#49123)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
23595f7fe5
|
fix(bedrock): backfill published regional service tier prices (#49113)
* fix(bedrock): backfill published regional service tier prices * fix(bedrock): remove the eu.anthropic.claude-haiku-4-5-20251001-v1:0 row --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
7add548171 | Merge origin/main into litellm_presidio_logging_only_mask_failure_logs | ||
|
yuneng-jiang
|
85af2f65c9
|
refactor(helm)!: connect both charts to a separately deployed Lens (#49169)
Neither chart bundles Lens anymore. The Lens subchart, its lock entry, the vendored chart archive, the templates/lens resources and the bundled-Lens install smoke are gone. A connection-only lens block wires the gateway and backend to a Lens deployed from its own chart, and an enabled lensWorker value fails the render instead of being ignored. Supersedes #47869, which made the same change for helm/litellm only. |
||
|
norkelbyte
|
44cd858697
|
fix(cost): bill the requested model's rates when the response names an unpriced model (#48900)
* fix(cost): bill the requested model's rates when the response names an unpriced model Fireworks answers chat and /v1/messages calls with a display name such as "Qwen 3.8 Max" in the response's model field. The cost path priced that name, the Fireworks calculator fell back to its free default bucket without raising, and the call billed $0 even though the requested model has a row. When the response model resolves to no cost-map row or rule that declares rates and the requested model does, price from the requested model. A priced response model still wins, so dated snapshots keep their own rates, and an unpriced requested model such as a router alias leaves the result unchanged. * perf(cost): check the exact cost-map row before generalization rules when picking the priced model * fix(cost): apply the priced fallback to the provider response model streaming records * perf(cost): skip the served model's rule lookup when the requested model has no rates * fix(cost): price the requested model without a model arg and keep Fireworks size buckets behind the proxy When completion_cost runs with no model, the requested model now comes from the call's hidden litellm_model_name, so a display-name response still bills at the requested model's rates. A Fireworks model whose name carries a parameter size counts as priced through its size bucket. The Fireworks per-token lookup now treats a cost-map row that prices nothing, like the empty row the Router registers for an unlisted deployment, as missing, so an unlisted Fireworks model bills at its size bucket instead of $0 even when Fireworks echoes its slug. The "row prices anything" helpers move to llm_cost_calc/utils so the Fireworks lookup can reuse them without a circular import. * fix(fireworks): keep prices from a matching fallback rule for an unlisted model An unlisted Fireworks model that only a fallback generalization rule prices has no raw cost map row, so the empty-row check sent it to the size bucket and dropped the rule's rates. The Fireworks lookup now keeps the model info when the row or a matching rule declares a rate. * fix(cost): price a bare response model under the requested provider and drop the unlisted Fireworks bucket change The logging payload picks the pricing model without a provider, so a bare echo like mistral-large-latest failed its price check and the requested alias took over model_map_key. The served name is now priced under the requested model's provider, and an echo that is the requested name minus its provider prefix skips the checks. Treating the Router's empty price row as missing let unlisted Fireworks deployments bill from a name guess, which misprices dedicated deployment ids and routes some short names to another provider's rates. That change and its tests go back to the merge base, so unlisted Fireworks models keep billing $0 behind the proxy as on main. * perf(cost): read the response model's price row without building full model info An unpriced display name like Qwen 3.8 Max made the price check build full model info, and Fireworks' supported params lookup scans the whole cost map for an unknown name. The row check now resolves the cost map key through the cached model info helper, so that pick takes about 0.45 ms instead of several. A regional test covers a Bedrock call whose response names an unpriced display name with no model passed, which bills the requested model's regional row. The duplicate get_model_info wrapper goes, since its twin already does the same thing. * fix(cost): keep the served model's filled-in and size-bucket rates, and cache misses A served row priced only by a fill_missing_for_providers rule bills those rates on main, but the price check read only the raw row and billed the requested alias instead. It now merges the rule's fill-in keys through the helper model info lookup uses. A served Fireworks or Together AI id that names its parameter count keeps its size-bucket price, as on main, instead of switching to the requested row. The served model is checked first, cost map misses are cached and cleared with the model info cache, and the rules are read only for a name that resolved through one. A priced snapshot behind an Azure deployment name takes about 19 us of CPU time against 7 us on the merge base, down from about 300 us. * fix(cost): bill the requested model's listed rates over a served size-bucket guess A served Fireworks or Together AI name that is not in the cost map but names its parameter count, like a display name with a size in it, now bills at the requested model's listed rates instead of the provider's size-bucket price. The bucket price still applies when the requested model has no rates, as on main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Gopal Kamineni
|
a7030a5a2c
|
fix(vertex_ai): batch Claude models in Anthropic's native row shape (#45715)
* fix(vertex_ai): batch Claude models in Anthropic's native row shape
Claude deployments on Vertex were filed under publishers/google/models/<claude>
when the batch input was uploaded, so create_batch asked Vertex for a Google
model that doesn't exist and got a bare 404 on batchPredictionJobs. The rows
were also written as Gemini GenerateContentRequests, which Vertex rejects for
Claude even with the right model path
Claude uploads now go under publishers/anthropic/models/<model> and each line
becomes {"custom_id", "request": <Anthropic Messages body with
anthropic_version vertex-2023-10-16>} using the existing Vertex Claude
transform. Output rows in the Anthropic message shape are converted back to
OpenAI batch rows (custom_id from the row, errors as failed rows), native rows
are costed from Anthropic usage, and create_batch fails fast with a 400 when a
Claude job targets the global location, which Vertex doesn't serve for
partner model batches
Fixes #45671
* test(vertex_ai): cover malformed Claude output rows and rows without custom_id
* fix(vertex_ai): build Claude batch rows with chat's param mapping and surface Vertex batch create errors
Claude batch rows now go through get_optional_params like Vertex Claude chat, so unsupported params, top_k, response_format and extra_body follow drop_params. Tool names Claude would rename get a 400 at upload. A refused batch create returns Vertex's own status and message.
* fix(vertex_ai): price Claude batch output rows without a request echo, keep @ version ids, refuse a json_tool_call user tool, keep Vertex error headers
* fix(vertex_ai): keep httpx.HTTPStatusError on Vertex batch create errors and apply deployment chat params to Claude batch rows
* fix(vertex_ai): keep Vertex's status code on batch create errors through the proxy
* fix(vertex_ai): keep Vertex's error body on batch create errors and require custom_id on Claude lines
* fix(vertex_ai): keep the old batch create error text and Google object names
A failed Vertex batch create keeps httpx's error line first and adds Vertex's body after it, and logs the status and body again. Only Claude object names keep the @ version, so Google models are stored under the same names as before
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
bap1106
|
7aa0393cb6
|
feat(providers): add CLF AI Gateway as an OpenAI-compatible provider with ten models (#39324)
* feat(providers): add CLF AI Gateway (clf_ai_gateway), OpenAI-compatible, 9 models OpenAI-compatible gateway at https://api.clfaigateway.dev/v1 serving open-weight models (GLM, Kimi, DeepSeek, Qwen) on Cloudflare Workers AI upstream. Follows the DeepInfra pattern: ClfAiGatewayConfig(OpenAIGPTConfig), CLF_AI_GATEWAY_API_KEY / _API_BASE, provider registered in enum, provider lists, endpoint inference, lazy-import registry, ProviderConfigManager, validate_environment. Supported params deliberately omit functions/function_call/logit_bias (gateway returns 400 for them). 9 cost-map entries (root + backup) with measured context/output limits and currently billed prices; endpoint-support entry; README row. Mocked tests only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(clf_ai_gateway): declare each model's real reasoning_effort levels The per-level supports_*_reasoning_effort flags cannot express the sets this gateway actually takes, so all nine models advertised a wrong one: every model offered "minimal" and eight offered "none", neither of which the gateway accepts, five dropped "xhigh", which it does accept, and qwen3.8-27b offered "high", which it rejects. That set surfaces through /model_group/info and the model picker, so a user picking a level off it gets a 400 from the gateway. Declare reasoning_effort_levels instead, taken from the gateway's own GET /v1/public/models, which resolve_supported_reasoning_efforts reads first and uses whole. Declaring it also stops clf_ai_gateway/deepseek-v4-pro and clf_ai_gateway/deepseek-v4-flash inheriting effort metadata from the unprefixed deepseek entries of the same name. * test(clf_ai_gateway): name the gateway-rejected params instead of commenting them * fix(clf_ai_gateway): list the provider in the proxy Add Model form main now requires every LlmProviders value to have an entry in provider_create_fields.json (or be frozen as unlisted), so the Add Model dropdown cannot silently miss a backend provider. Add the clf_ai_gateway entry in the same shape as the SCX.ai registration: required API key, optional API base with https://api.clfaigateway.dev/v1 as placeholder, and a cost-map model as the default model placeholder. A provider test pins the entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(clf_ai_gateway): sync prices and reasoning_effort levels with the live catalog CLF AI Gateway raised its launch discount from 40% to 60% off list on 2026-09-26. It also re-measured every model's reasoning_effort levels against the current Workers AI behaviour. - input/output/cached per-token prices for all 9 models now match GET https://api.clfaigateway.dev/v1/public/models to the nano; qwen3.8-27b cached input also follows the upstream cached rate published on 2026-09-25 - reasoning_effort_levels: add "max" to deepseek-v4-flash, deepseek-v4-pro, glm-5.2 and glm-5.3-flash; add "none" to kimi-k2.6 (it really disables thinking there); drop "none" from glm-5.3 (upstream runs it as "max") Context windows, output limits and vision flags are unchanged and were checked against the same endpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(clf_ai_gateway): drop a rebind-ok that no longer suppresses anything After rebasing onto main, the type-discipline gate's new LIT013 rule flags the `# rebind-ok` on the endpoint-inference `dynamic_api_key` assignment. The rebind rule no longer fires on that line, so the suppression is inert. The sibling endpoint branches carry no comment there either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(clf_ai_gateway): expect the re-measured reasoning_effort levels Follows the catalog sync: max on deepseek-v4-flash, deepseek-v4-pro, glm-5.2 and glm-5.3-flash, none on kimi-k2.6, and no none on glm-5.3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(clf_ai_gateway): move the provider tests to tests/unit/llms main moved every per-provider test from tests/test_litellm/llms/<provider>/ to tests/unit/llms/<provider>/ (DeepInfra lives at tests/unit/llms/deepinfra/). This branch still added the old path, so no CI shard claimed it. That failed assert-ci-coverage and assert-shard-coverage, and the file never ran (hence the low codecov/patch). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(clf_ai_gateway): assert routing, billing, and dropped params instead of pinned gateway facts * fix(clf_ai_gateway): keep the type-discipline gate at its merge-base count * chore(clf_ai_gateway): regenerate provider endpoint support * ci(clf_ai_gateway): run the provider tests in the llms unit shard * fix(clf_ai_gateway): drop the retired max_tokens field from the cost map rows * fix(clf_ai_gateway): forward params the gateway accepts, drop undeclared reasoning levels, add deepseek-v4.1-flash * fix(clf_ai_gateway): drop the unreachable non-chat supported-params branch * fix(clf_ai_gateway): send a dict reasoning_effort as its level * fix(clf_ai_gateway): refuse a reasoning_effort dict without a level unless drop_params drops it --------- Co-authored-by: bap1106 <bap1106@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
tin-berri
|
fd594ab0cf
|
fix: enforce LiteAdmin MCP route permissions (#48080)
* fix(admin-mcp): expose and enforce credential route permissions * fix: restore beta header and telemetry test parity * fix(admin-mcp): align permissions with registered HTTP routes * test(admin-mcp): verify encoded paths through HTTP authorization * test(admin-mcp): simplify HTTP permission regression fixtures |