mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-11 08:38:24 +00:00
740 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
b89fb13d51
|
fix(register_model)!: keep bedrock geo deployments priced across price data reloads (#41523)
* fix(register_model): keep bedrock geo deployments priced across price data reloads
A request-time pricing registration inserting into litellm.model_cost while a
price data reload walked it could abort the reload (dictionary changed size
during iteration in add_known_models), or trip the lookup map rebuild inside
get_model_info so the deployment replay fell through to the unmapped path and
wrote a raw bedrock/au.anthropic.<model> entry with no token pricing, a $0
shadow re-rolled on every later reload. Walk snapshots of the map, bind the
case-insensitive lookup map locally, and let the unmapped path inherit the
whole built-in entry of a provider get_model_info would accept.
* fix(register_model): keep synthesized zero costs out of inherited built-in pricing
An unmapped alias that inherits its built-in entry now takes only the token
prices the catalogue really carries, so a synthesized zero never lands next to
the caller's custom price. The router regression test reports the writer's
failures instead of swallowing them, and the local names carry Final.
* fix(register_model): match inherited built-ins by the key provider prefix when the value names none
* fix(utils): resolve a tripped lookup from the key's own catalog row before its region sibling
* fix(register_model): keep custom flat prices above inherited tier rates
* fix(register_model): keep the other direction's tier rates when a custom flat price covers one
* fix(register_model): keep a catalog row's tiers when its lookup trips, and snapshot request-path cost map walks
A tripped get_model_info on a key that is itself a catalog row now inherits that row whole, so a custom flat price bills a long prompt at the row's above-threshold rate on both lookup paths instead of flipping until the next reload. The fal_ai, fireworks, ollama and gpt-5 request-path walks over litellm.model_cost iterate a snapshot so a concurrent reload cannot raise mid-request.
* fix(register_model): price route-prefixed geo keys on a tripped lookup and replace cost map entries atomically
* fix(register_model): move every alias of a cost map entry to its registered price
* perf(register_model): read alias names from the shared entry instead of scanning the cost map
Each registration scanned every cost map entry to find the names sharing it,
about 1 ms on the 4,189-entry map, and a proxy request to a custom-priced
deployment registers twice. Aliased entries now carry the names they are
served under, so a registration reads them directly, and an alias the caller
pointed at its own entry keeps that entry. The tests this PR added lose their
docstrings
* fix(utils): keep capability flags and write through on a tripped lookup in register_model
A key the lookup cannot map now seeds its registered entry from the capability
rules its unregistered lookup resolves to, so registering a region alias no
longer turns off supports_system_messages and similar flags. A lookup that
trips mid-walk on a key naming a catalog row writes into that row instead of
leaving a shadow key that goes stale. Adds regression tests for both and for
the request-path snapshot walks.
* fix(utils): retry a tripped registration lookup and rebuild a registration a reload races
A lookup a concurrent write trips is retried instead of writing into the
catalog row its key names, so a per-request price never reprices a shared
row. The inherited entry leaves the matched row's provider behind, so a
provider-less azure_ai key stays priced, and a registration whose map a
reload swapped mid-build is rebuilt from the new map.
* fix(utils): write each registration into the map it was built from and rebuild it when a reload replaced that map
* test(integration): price data reload keeps every registered deployment priced
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit
|
||
|
Kerry Lu
|
6df738c397
|
fix(anthropic): apply tool_config cache-control points on chat and /v1/messages (#47394)
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
(cherry picked from commit
|
||
|
Kerry Lu
|
c2788016b7
|
fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists (#47373)
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
ef447d8d3e
|
fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards (#46102)
* fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards
* test(rust): check the dangerous-tool-use beta reaches the wire with safeguards
* fix(rust): derive the dangerous-tool-use beta only for hosts with a beta map column
DeepSeek and the other hosts without a column in anthropic_beta_headers_config.json
keep forwarding safeguards and client betas as sent. Adds the wire-level audit tests.
* fix(rust): add the safeguards beta on Azure AI and Claude Platform hosts
Azure AI and Claude Platform Messages transforms now add the
dangerous-tool-use beta for a non-null safeguards field, through the
Azure AI and Anthropic beta-map columns. The safeguards wire and chaos
tests get unique file names so they do not collide with the Python
fix's tests, the Python-route test asserts the fixed behavior behind a
BUG skip, and the DeepSeek test checks that no beta is added
* fix(rust): keep Claude Platform out of the safeguards beta until a route can prove it
Claude Platform has no beta-map column and no QA account, so it goes back to sending
safeguards with no derived beta, like DeepSeek. Also release held chaos-test requests
before the thread pool waits on them, and mark the no-column test as rstest.
* fix(rust): derive the safeguards beta for Claude Platform and keep the null-cell test off Databricks
Claude Platform on AWS answers 400 to safeguards without dangerous-tool-use-2026-09-03 and 200 with it, so its Messages transform derives the beta through the Anthropic column again. The null-cell case now uses only Bedrock Converse, since Databricks support for the beta is changing in a sibling PR.
* fix(rust): drop a null safeguards key instead of sending it without its beta
Anthropic rejects any safeguards key, null included, when the
dangerous-tool-use beta is missing, so an explicit "safeguards": null on the
Rust gateway still answered 400. The shared Messages transform now drops a
null safeguards key on every host, as the proxy already drops None params
before calling Rust, so it answers 200 with no beta.
The scripted peer in the integration wire tests now rejects the key on
presence, matching the real API.
* test: cover Bedrock Mantle in the null safeguards case and match the vendor's non-list wording
The Rust null-safeguards regression test now also runs for bedrock_mantle, whose beta is keyed on the field being present. The scripted Anthropic peer answers a non-list safeguards value with the real API's "valid array" wording.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
5636e41a81
|
fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards (#43955)
* fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards
* test(azure_ai): pin the outgoing Foundry request keeps the dangerous-tool-use beta next to safeguards
* test(azure_ai): inject the Foundry client and scope the beta cache reset to the test
* test(azure_ai): own and close the mock transport client, drop the docstring entry
* test(integration): audit the safeguards beta derivation across anthropic-compatible providers
* fix(github_copilot): keep safeguards from deriving the dangerous-tool-use beta
* test(integration): kill the chaos victim worker through the harness helper
* fix(anthropic): derive the dangerous-tool-use beta only where the beta map supports it and honor additional_drop_params
The safeguards beta is now added only for providers whose beta map column accepts it, so github_copilot, Eden AI and the OpenAI-compatible JSON providers keep their base wire. additional_drop_params now runs before the provider headers are built, so a dropped param no longer drives its derived beta.
* fix(anthropic): load the beta map off the event loop and pin a dropped context_management
The first /v1/messages request that needs the remote beta map now loads it in a worker thread,
so a slow fetch no longer stalls every other request on the worker. An integration cell pins that a
deployment dropping context_management sends neither the field nor its derived beta.
* fix(anthropic): load the beta map only for safeguards requests, once per worker
* fix(anthropic): gate safeguards on the bundled beta map and load the remote map on its own thread
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit
|
||
|
yucheng-berri
|
c4f545cf7b
|
fix(guardrails)!: backport #45874 to rc/1.107.0, preserve Bedrock Converse redactions and honor tool-only scans (#47234)
Some checks failed
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Applies #45874's diff at
|
||
|
yuneng-jiang
|
6a0603e00f
|
feat(count_tokens)!: backport the count_tokens stack to rc/1.107.0 (#46839, #46843, #46583, #47057, #47064, #47163, #46777) (#47209)
* fix(proxy)!: count prompt-only token_counter requests with the provider when call_endpoint is set (#46839) * fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set * fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails * fix(proxy): log a prompt-only token count fallback as a one-line warning * fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI * fix(proxy): count a token_counter prompt sent next to an empty messages or contents list * fix(proxy): count a prompt next to messages on the local token count instead of answering 500 * fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form. * test(proxy): count tools with a prompt in the token counter fallback test * test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests * test(proxy): name the token counter fallback test by what it checks (cherry picked from commit |
||
|
yucheng-berri
|
9445634193
|
fix(bedrock): backport #46307 and #46524 to rc/1.107.0 (#47157)
* fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307) * test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Backport of #46307 to rc/1.107.0 Cherry-picked from |
||
|
yuneng-jiang
|
ab58c55d4b
|
fix: backport the third batch of rc/1.107.0 CI triage fixes from main (#47174)
* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy (#47152) * test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy These two files register deployments with their own model settings on the gateway. They now boot a module-scoped proxy on a scratch database, so their setup cannot affect other tests that share the worker's proxy * test(integration): seed the isolated gateway with the same router settings as the worker gateway (cherry picked from commit |
||
|
yuneng-jiang
|
0c859e592f
|
fix(tests): drain SDK logging before restoring callbacks (#46924) (#47143)
(cherry picked from commit
|
||
|
yuneng-jiang
|
4bf1633452
|
fix!: backport the second batch of rc/1.107.0 CI triage fixes from main (#47090)
* fix(bedrock)!: block realtime replay after an input-first start and reject malformed Nova Sonic session settings (#46936) * fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice * test(bedrock): name the malformed realtime setting test for its string cases * fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params. * fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats * fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices * test(bedrock): cover realtime replay, refusal and committed failure through a real proxy Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again (cherry picked from commit |
||
|
yuneng-jiang
|
a00d8a6ee1
|
fix(proxy): backport the 1.107.0 rc triage fixes to rc/1.107.0 (#47043)
* fix(proxy): give /team/delete transactions a 30s timeout (#46979) Both interactive transactions in /team/delete used Prisma's 5s default. A lock wait or a short database stall past 5s expired the transaction after the keys were already deleted, so the call returned 503 and left the team row, memberships and team models behind (cherry picked from commit |
||
|
武本龍門
|
43be1c2ae7
|
fix(router)!: keep RPM, region, and param checks when token counting fails (#44544)
* fix(router): stop skipping rate-limit checks when token counting fails Router._pre_call_checks() caught any token_counter() exception and returned the full, unfiltered deployment list, bypassing the RPM, region, and invalid-params checks for every deployment - not just the context-window check that actually depends on the count. A content block token_counter() cannot parse (e.g. an invalid image_url detail value) silently disabled rate-limit enforcement for that request, with only a log line and no error surfaced to the caller. Now a counting failure only disables the context-window check; the other checks still run for every deployment as before. Input-token resolution is pulled into its own method and computed once before the loop, so nothing needs to track "did counting fail" with a mutable flag across iterations. * test(router): cover the async region check when token counting fails Replace the unit test of the private input-token resolver with a regression test through async_get_healthy_deployments, the path the proxy uses: an EU-pinned request whose content cannot be token-counted must not get the US deployment * test(router): cover the param filter when token counting fails and satisfy router coverage * fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params, drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away instead of a retried 429. Deployment context windows are resolved once per call. * fix(router): keep the retried 429 when a cooling-down deployment takes the param * fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks * fix(router): steer by deployment capability, apply drop settings after region and RPM checks A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing * test(router): make the github_copilot pre-call filter test hermetic in CI * test(router): cover pre-call region and response_format filters through the proxy * fix(router): judge the 400 against the request's own deployments, not a name lookup * fix(router): judge the 400 against the passed deployments when the group name lists none * fix(router): let a request's drop and allow settings replace the deployment's in the param check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Aryan Pardeshi
|
a844dadcd2
|
feat(gemini): allow disabling thought summaries while keeping thinking on (#36460)
* feat(gemini): allow disabling thought summaries while keeping thinking on Both reasoning-effort mappers hardcoded includeThoughts to True whenever thinking was on, so thinking ON with summaries OFF was unreachable even though that is the google SDK's own default. Summaries are the narration rather than the reasoning, and for callers that discard them they are pure added latency. Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing changes for anyone who does not set it. It gates the includeThoughts=True branches in the thinkingBudget mapper, the thinkingLevel mapper, and the anthropic-style thinking param mapper. The disable and none paths still force includeThoughts=False: with thinking off there is nothing to summarise. * test(gemini): assert thought-summary flag through the outbound Gemini request * feat(gemini): let a request's thought summary ask override the gateway default thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing * feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default * fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag Normalize gemini_include_thoughts_default through normalize_drop_params so a YAML or env value written as "false" or "0" turns thought text off and a null or unknown value keeps the default on. thinking.display "omitted" no longer counts as a per-request ask, so the gateway flag decides, matching /v1/messages. Adds proxy integration cells for chat, /v1/messages and /v1/responses that prove the flag, every per-request ask, streaming, and a burst across workers. * test(gemini): match the streamed Responses usage, which omits null token details * feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks * fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply * fix(caching): key the response cache on the summary alias wherever the request carries it The cache key now reads reasoning_summary and reasoningSummary through the same reader as the summary fold, so an alias inside extra_body gets its own key too * test: type the new summary-ask unit tests and mark their locals Final * test: annotate the remaining summary-ask unit tests the PR touches --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
debanshd
|
1b1ddef1b2
|
fix: preserve non-null anyOf branches when stripping null types for Vertex (#41683)
* fix: preserve non-null anyOf branches when stripping null types for Vertex * refactor: build anyOf branches immutably and drop explanatory comments * style: apply ruff format * fix: annotate anyOf list construction for the LIT002 budget * fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising * test(vertex_ai): pin empty anyOf passthrough * fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold * fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing * fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4e49394ade
|
fix(anthropic): keep trailing user messages with None or empty list content (#41397)
* fix(anthropic): keep user messages with None or empty list content so requests do not end on an assistant turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid mutable empty-list construction in sanitizer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid rebinding sanitizer input Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): keep a trailing user turn whose content key was dropped litellm.completion strips None fields from every message before the provider transform, so a user message sent with content null reaches the Anthropic converter with no content key and raised KeyError. Read the content with get so that turn gets the same placeholder as an empty list. * fix(anthropic): only fill user turns with empty content and keep their cache_control * fix(anthropic): fill each empty user message in place like an empty string * fix(anthropic): add the placeholder only for an empty turn or a cache breakpoint, in linear time * test(anthropic): cover the tool-result guard with cache_control and widen the linear-time bound * test(anthropic): cover empty user turns through the proxy on chat, responses and messages * test(anthropic): type the empty user turn tests and move the linear-time check to a benchmark --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
f281f77bc0
|
fix(guardrails): record every content filter match in logging_only and offer the mode in the UI (#46673)
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test * fix(guardrails): record every content filter match in logging_only and offer the mode in the UI * style(tests): format the content filter logging_only integration test * refactor(guardrails): keep the logging_only predicate and scan override within the type gates * fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning * test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test * test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects * refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach * test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter * test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up * test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
a3251a3bfd
|
fix(responses): preserve provider request IDs in streaming failure logs (#45453)
* fix(responses): preserve provider request IDs in streaming failure logs * fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams Bedrock models that run through the chat-completions bridge (Claude, converse, invoke) raise in-stream failures from the event-stream decoder as a BedrockError without headers, so their failure logs kept error_provider_request_id null. The decoder now passes the stream's request-id headers to those errors. Both the native Responses iterator and the decoder pass only the PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs pick up the request id and none of the 200 response's other headers. MidStreamFallbackError gets its response at construction instead of mutating it afterwards. --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
b21e659c24
|
fix(proxy)!: count endpoints return errors in their source API shape instead of silent local counts (#46815)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system * fix(proxy): count endpoints return missing-key and request errors in their source API shape * fix(proxy): drop imports the count error change left unused * fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope * fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures * fix(proxy): count with the global OpenAI and Anthropic keys chat already uses * fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway * fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response * fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads * fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses * fix(proxy): mint the OpenAI workload identity bearer inside the count error handling * fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401 --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Michelangelo Setaro
|
c288dc7da0
|
fix(router): let an exact base_model widen a deployment's context window (#46197)
* fix(router): honor explicit bedrock base model capacity * fix(router): honor an exact base_model's capacity for every non-Azure provider Extends the Bedrock-only check to every non-Azure provider and only uses the base_model when it has its own exact cost-map entry, so an unmapped or wildcard base_model falls back to the route id instead of raising in the router. * test(router): call _maps_to_exact_model_info directly in its cost-map tests * fix(router): let an exact base_model only widen a deployment's context window A base_model used as a pricing stand-in with a smaller window no longer narrows the route's limit, and a route with no window never takes one from base_model * test(router): count the base_model window helpers as covered through get_router_model_info * fix(router): widen only from a model_info base_model naming the same model A stand-in base_model for a different, larger model widened a smaller deployment's window, so pre-call checks sent it prompts the provider rejects with 400. The window now widens only when model_info.base_model names the same provider model id as the route, which is also the field /model/info reads. * fix(router): match a base_model's model id across Bedrock region prefixes and @ qualifiers * fix(router): read litellm_params.base_model too and look the base row up as written * test(router): cover a same-model base_model window on chat, messages, responses, and complexity escalation --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
cda19d6a29
|
test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846)
The operator trace wait stopped at the root and redis spans, but the gen-AI span is closed later by the async success callback. When it had not landed yet, the tenant mirror was built without it and the gen-AI assertion failed. Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
b5870df967
|
fix(bedrock-passthrough)!: return AWS-shaped errors so AWS SDKs read the code and message (#46522)
* fix(bedrock-passthrough): return AWS-shaped errors so AWS SDKs read the code and message A guardrail block on a /bedrock route now answers with x-amzn-errortype ValidationException and a top-level message, and an error Bedrock returned (such as a 429 ThrottlingException) is passed through with its status, x-amzn-errortype and body * fix(bedrock-passthrough): keep 5xx guardrail failures and failure-hook replacements unchanged * test(bedrock-passthrough): guardrail vendor failures and unknown models keep the litellm error * fix(bedrock-passthrough): only answer guardrail exceptions that blocked content as ValidationException * fix(bedrock-passthrough): pass Bedrock's content-type through with its error body * refactor(bedrock-passthrough): wrap the guardrail ProxyException instead of copying its fields Unit tests drive the error response with the real request processor and ProxyLogging * test(bedrock-passthrough): cover the direct model route, invoke, and auth, budget and rate-limit errors * fix(bedrock-passthrough): pass through only errors a Bedrock model operation returned A guardrail service's HTTP error that carries x-amzn-errortype keeps the litellm error * refactor(bedrock-passthrough): mark Bedrock's errors where the passthrough raises them instead of matching the URL A guardrail service's HTTP error never goes through the passthrough call, so it keeps the litellm error whatever its URL --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
79ca820933
|
fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
devin-ai-integration[bot]
|
91ae7d1b7d
|
fix(cost): bill Responses API image generation and usage reported web search as tool usage cost (#41877)
* fix(cost): bill Responses API image_generation_call tool usage Price completed image_generation_call output items from tool_usage.image_gen token counts when present, otherwise per image from the quality and size on the item, with the image model taken from tools[].model (default gpt-image-1). Add the result into tool_usage_cost alongside web search and file search, and read tool_usage from the top level or usage.tool_usage. Squashed and rebased onto main * style(cost): annotate read-only views and mark casts for the type discipline gate * fix(cost): price Responses image tool from tools[].model only, no default model or provider * refactor(cost): drop redundant status check, unread usage fields and unused output_type literal * test(cost): integration case for Responses image_generation tool token billing --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
dba86bec28
|
fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them (#46767)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Kerry Lu
|
5ed3629242
|
fix(responses): write one spend-log row per native WebSocket turn (#46768)
* fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn * test(responses): mock the websocket slot release and mark the per-turn start time naive * style(responses): keep the per-turn start time line within ruff format * refactor(responses): type websocket turn events as read-only mappings * fix(responses): settle the websocket budget reservation on the first turn only Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly * fix(proxy): keep the legacy limiter's websocket slot until disconnect A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
cf1e6b19ab
|
fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache (#46807)
* fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache SAML login tickets, the single-use assertion mark and cached IdP metadata lived in user_api_key_cache, which only gets Redis when enable_redis_auth_cache is on. With several replicas or workers the IdP callback often landed on a process that never saw the ticket and got a 401, and IdP-initiated sign-in was refused even with Redis configured. SAML state now has its own DualCache that _attach_redis_usage_cache always wires to Redis, like the CLI SSO session cache, and the SAML cache calls are async. * test(sso): drop docstring from the SAML proxy pair fixture * test(sso): cover the SAML login and metadata endpoints |
||
|
Mateo Wang
|
d25b58f744
|
fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
yucheng-berri
|
627bcab42b
|
fix(guardrails): honor streaming settings on the Noma v2 guardrail (#46606)
* fix(guardrails): honor streaming settings on the Noma v2 guardrail The Noma v2 initializer now passes streaming_buffer_until_moderated, streaming_end_of_stream_only, streaming_sampling_rate and streaming_buffer_release_on_scan from the guardrail config to NomaV2Guardrail, so the unified streaming hook applies them. The constructor validates them, so a custom guardrail class built on NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults, and an invalid value is ignored with a warning naming the guardrail, so the guardrail still loads. Resolves LIT-9633 * test(guardrails): cover Noma v2 streaming settings across clients, failures and workers Adds integration cells for the custom-subclass loader, string and invalid values, unset and false keys, a Noma outage with and without block_failures, an upstream rejection, n=2 choices, pre_call and non-streaming guardrails, the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a PATCH while traffic flows, a burst while Noma is down, a slow Noma, and replaced workers. Resolves LIT-9633 * test(guardrails): pin the PATCH probes to one worker at a time Each probe suspends the other worker and opens a fresh connection, so the test checks that every worker applies the patched setting. Resolves LIT-9633 * test(guardrails): check sampled scans by count, not window position /v1/messages hands the streaming hook raw network reads, so where the default sampling windows fall depends on timing; the base leg caught one shifted window. Check the scan count and the final full-text scan instead. Resolves LIT-9633 --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
250e6bd24b
|
test(integration): build OpenAI and Anthropic SDK models before tests start threads (#46797)
K2 in test_logging_only_scope_chaos.py flaked on CircleCI with AttributeError: type object 'BaseModel' has no attribute '__pydantic_core_schema__' raised inside the OpenAI client while its 30-thread control burst parsed responses The SDKs define their response models with deferred schemas, so the first parse of each model rebuilds it, and pydantic's model_rebuild is not safe when several threads hit it at once. Every integration worker now builds those schemas in pytest_configure, before any test can start a thread pool Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
b47f7a4712
|
fix(responses): preserve explicit websocket API keys (internal copy of #40853) (#46724)
* fix(responses): preserve explicit websocket API keys Refresh the one-line fix from BerriAI/litellm#34253 against the current default branch Co-authored-by: puffer <389053104@qq.com> * test(responses): verify websocket authorization Cover explicit deployment keys, precedence over global keys, and the existing global fallback at the upstream WebSocket transport boundary * test(responses): type the authorization capture Replace the mutable header list with a typed, final Future carrying the upstream authorization value, addressing the review on #40853 * Refactor test_openai_websocket_authorization method * fix(responses): resolve websocket keys with the deployment's provider Pass the deployment custom_llm_provider to get_llm_provider so an api_base host never swaps in another provider's env key, and drop the OpenAI-only global key fallbacks so non-OpenAI websocket deployments resolve their own env keys the way the HTTP route does * fix(responses): let a deployment's own websocket key beat env keys A deployment with an api_base on a host litellm recognizes, and no declared provider, got that host's env key ahead of its own api_key. The explicit key now always wins, matching the HTTP Responses route. Drop the litellm_params api_key fallback, which could never be set because api_key is a named argument * fix(responses): keep openai deployments on other providers' known hosts on the HTTP stream An openai deployment whose api_base is a known OpenAI-compatible provider host (DeepSeek, Groq, ...) opened a native Responses WebSocket there and got a 405, because those hosts serve /responses over HTTP only. Route it through the managed HTTP stream with the deployment's own key instead. * fix(responses): resolve openai deployments on known provider hosts as that provider An openai deployment whose api_base is a known provider host (DeepSeek, Mistral, ...) now resolves to that host's provider on the Responses WebSocket again, as before this branch, so it keeps that provider's chat completions bridge, pricing and env key fallback. Its own api_key still wins over the env key. The known-host test now mocks the upstream HTTP route and checks the route, the bearer and response.completed. * test(responses): prove websocket deployment keys and declared providers reach the provider socket * test(responses): use the reported bare model shape for the websocket deployment key cell --------- Co-authored-by: Ysh <734645956@qq.com> Co-authored-by: puffer <389053104@qq.com> |
||
|
mrinal-berri
|
bd60cc8608
|
fix(bedrock): forward Nova Sonic realtime tool calls with their arguments (#46214) | ||
|
devin-ai-integration[bot]
|
67bfb7a148
|
fix(ollama): prevent reused response ids from dropping spend logs (#39227)
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): narrow the provider response id with isinstance instead of cast Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): validate callback fields for current type gates * test(gcs_pubsub): assert provider response id in spend metadata * fix(ollama): normalize reused response ids automatically --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
b602209ca6
|
test(integration): hand every harness server a port from one per-worker block (#46761)
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit
|
||
|
Yassin Kortam
|
1e5d83f6ce
|
ci: run mcp interactions and redis cluster iam integration tests only in circleci (#46780)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
77c22a4c01
|
feat(proxy): add agent_access_default_deny for least-privilege agent access (#43979)
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align A2A model visibility assertions with supported routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep A2A model entries on their existing routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): point the listing model at the harness upstream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): coerce require_key_agent_access_defined like the settings validator Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): inject agent access lookups instead of patching the handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
440744f967
|
feat(mcp): enable verified stateless protocol capabilities (#46264)
* feat(mcp): advertise verified stateless capabilities * fix(mcp): honor rate limits during capability discovery * test(mcp): cover protocol interactions and scoped enablement rollback * fix(mcp): preserve legacy capabilities and bound discovery probes * fix(mcp): keep discovery results single-assignment * ci(mcp): distribute acceptance cases across existing workers * ci: balance MCP integration cases across idle workers * ci: isolate MCP interaction acceptance workload * test(mcp): acknowledge fixture notifications after dispatch * test(mcp): drain session dispatch before fixture deletion * test(mcp): preserve deletion of streaming fixture requests --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d489c04554
|
fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking (#46741)
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking * fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds Bedrock Invoke copied every caller thinking key when adding display updates, so an unknown key still reached the provider with drop_params on. Filter that copy the same way, and cover each chat call site plus the global litellm.drop_params path in tests * fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params * test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke |
||
|
Mateo Wang
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
Kerry Lu
|
8e92a47ff3
|
test(integration): assert the streamed usage.cost matches the spend row on every cost case (#46762)
* test(integration): assert the streamed usage.cost matches the spend row on every cost case * test(integration): type the streamed usage frames as JsonValue * test(integration): only require streamed usage.cost when the client received a usage frame --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
84ff9f84a9
|
fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses (#46720)
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses * fix(ui): send detail on playground Responses image parts and code snippets * test(messages): cover image detail on the Responses bridge wire |
||
|
Fede Kamelhar
|
68759a239f
|
fix(oci): drop non-function tools under drop_params and gate reasoning_effort on declared levels (#40732)
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it Fixes #31449. Two OCI GENERIC request-building defects that made Grok-on-OCI unusable for tool-using coding-agent clients (Codex over the Responses API, Claude Code over the Anthropic Messages API): 1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only supports function tools")` for any tool whose type is not "function", aborting the whole request when a client mixed a built-in/server tool in with its function tools. Non-function tools are now skipped with a warning; if none remain, `tools`/`toolChoice` are omitted from the request instead of sending an empty array. 2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere model, but OCI rejects it for all served xAI Grok models (`Model grok-4.20 does not support parameter reasoningEffort`) and for OpenAI's non-reasoning models (`Unrecognized request argument supplied: reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-* and Meta/Google models accept it. `_model_supports_reasoning_effort` encodes that, `get_supported_openai_params` stops advertising the param for those models, and `map_openai_params` follows the standard unsupported-param contract (dropped under drop_params, clear error otherwise) so `drop_params` / `additional_drop_params` now work as documented instead of the 400 surfacing from the service. Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model. The two existing reasoning_effort normalisation tests move from a Grok model to openai.gpt-5, which is the case they exercise. 21 new tests; the OCI suite is 406 passed. * fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools Review follow-ups on #40732: - `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag of the model's `oci/` catalog entry through `get_model_info` instead of hardcoding xAI / gpt-oss branches. Catalogued models without the flag (xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent from the catalog keep the historical pass-through. - Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b` (pricing from Oracle's public price list, 128K context per OCI docs). - `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only. OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output cap silently disappears) and Llama rejects it with 400, so the catalog's reasoning flag must not drive that switch. - After tool adaptation, a `FUNCTION` tool_choice naming a tool that was skipped (built-in / server tools) is dropped with a warning instead of being sent to OCI, which rejects the unmatched choice. Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a forced built-in now completes and still calls the function tool; Gemini and gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under drop_params; gpt-5 keeps maxCompletionTokens. * style(oci): use an immutable set for the tool-name lookup Keeps the type-discipline gate (LIT002, mutable collections) within its budget; the lookup is read-only. * fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive Review follow-ups on #40732: - When tool_choice forces a function that is not among the adapted tools (a skipped built-in or a misspelled name), send `toolChoice: NONE` instead of removing the choice. Falling back to the service default would let the model call any other supplied function even though the caller forced a specific one, which an agent processing untrusted content must not do. - Move the #31449 regression tests into the mapped test module (`test_oci_chat_transformation.py`) instead of a separate file. Verified live against OCI GenAI: Grok and Llama with a forced skipped built-in now complete with no tool call; a forced kept function is still honoured. * test(oci): point reasoning_effort comments at the in-module gating tests The referenced module was folded into this file. * fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels Non-function tools now follow the unsupported-param contract: a 400 naming drop_params without it, and dropped with a warning under it, with a forced dropped tool turning tool calls off. reasoning_effort is gated on the model's reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop sending a param the service rejects * fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it * fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name * fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name * fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap * fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR Under drop_params, non-function tools still drop when function tools remain and tool_choice does not force a dropped tool. A request whose every tool is dropped, or whose tool_choice forces one, now answers 400 instead of a tool-less reply the model can fill with invented results, or a swap to another tool. The maxTokens routing change for OCI Gemini goes back to base, since it is outside this fix and an open PR carries it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Yassin Kortam
|
82977e2465
|
fix(mcp): accept validated JWT subjects after virtual-key mapping (#46549)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Dushyant Acharya
|
06efa5de85
|
fix(proxy): honor disable_end_user_cost_tracking in spend logs and end-user tables (#40945)
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs * fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off * fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off * test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Yassin Kortam
|
d113e4a0dd
|
fix(proxy-extras): hold only Prisma's lock key during migrate deploy so it never waits on the request log index build (#46660)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
1ac9cb2a1d
|
fix(proxy-extras): accept unencoded DB passwords in the v2 migration lock and redact its errors (#46592)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
62af39aaf8
|
fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it (#46591)
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it With modify_params on, a trailing tool search turn whose call has a kept result got a placeholder answer from the factory and went out as a marked tool_use, but the cache control hook counted marks before that placeholder existed and skipped the mark, so a request at the cap reached Anthropic with 5 marks and failed with a 400. The hook now finds client answers on the sanitized messages, the same list the factory reads. Also make the replay checks cheaper (prefix check before validation, early return without provider_specific_fields, and look up client answers only when a tool call carries cache_control), drop the two audit cells that cover old gaps outside the tool search fix, and derive the pass-through test timeout from the graceful stop budget. * fix(anthropic): keep the duplicate tool result warning out of the cache control hook The hook finds client answers with only the step of the modify_params sanitize pass that pairs tool calls with their results, so a request with a duplicate tool result logs the warning once again instead of once per mark count. Also restore the Snowflake non-stream and cached-stream tool search cells as known bugs with pytest.skip instead of deleting them, and put the Anthropic pass-through test timeout back to 240 seconds. * fix(anthropic): count cache marks next to a message the pairing step cannot read With modify_params on and a marked tool call, the cache control hook paired tool calls with their results before counting. A message without a get method (a stray string or null in extra_body.messages, or an openai SDK ChatCompletionMessage) made that pairing raise AttributeError before any provider call, where main counted the marks and let the request through. The hook now pairs only when every message can be read the way the pairing step reads it, and otherwise counts on the messages as sent, like main did. * fix(anthropic): count cache marks after placing a later system message the way the request builder does * test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census * test(anthropic): align the tool search replay cells with main and cover a system message after the search turn The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882. The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its mark, and a new cell covers a system message after the trailing search turn in both stream modes * fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request * test(anthropic): cover marked tool calls next to a malformed tool id and a later system message * perf(anthropic): look up client tool answers only when a marked tool call is a server tool call Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count marks * test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell * perf(anthropic): place mid-conversation system messages in linear time for models that keep them * fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time * test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn * fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag * test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router * fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure * test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement * fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route. * test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse * fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow * test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one |
||
|
mgeorgaklis
|
b3c9d4c1f7
|
fix(prompt_templates): do not mutate messages in-place in function_call_prompt (#41676)
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt * refactor(test): avoid local list mutation and satisfy type discipline gate * style(prompt_templates): keep line length under 120 chars for ruff format * style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401 * style(prompt_templates): eliminate basedpyright errors in function_call_prompt * fix(prompt_templates): keep tuple and None system content when adding the function prompt Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged * fix(prompt_templates): reject unreadable system content without echoing it Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt * fix(prompt_templates): leave non-list system content failing as before None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated * fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params * test(prompt_templates): pin the function prompt on every system message * test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
835038cbbc
|
fix(websearch_interception): intercept the bare web_search tool type on chat completions (#46736)
* fix(websearch_interception): intercept the bare web_search tool type on chat completions * fix(websearch_interception): match the bare web_search type only on chat completions /v1/messages keeps its existing detector, so a non-Anthropic model there no longer short-circuits a bare web_search tool into raw search results * test(websearch_interception): cover the bare web_search type on chat and through the messages adapter |
||
|
yuneng-jiang
|
46913be92b
|
test(integration): wait on the right signal in five flaky integration contracts (#46737)
* test(integration): wait on the right signal in five flaky integration contracts The Prometheus shared-label tests waited for the latency metric and then read the request counter, which the logger writes after an await. They now wait for the counter The auto-router session poll returned on the first 200 while the caller expected two turns, so it now waits for the expected turn count The S3 queue cap test let its failing sink recover on a 15s clock that could expire mid-flush. The sink now stays down until a flush fails after every event is queued K3 normally takes 168-175s against a 180s limit, so the limit is now 300s The MCP env var browser spec reused an idle keep-alive socket that the proxy closes after 5s, so its API calls now send Connection: close * test(integration): add until_asserted so a wait polls the exact asserts it guards eventually() takes a predicate separate from the asserts that follow it, which lets a test wait on one value and then assert on another written later. until_asserted reruns a block of asserts until they all pass and re-raises the last failure on timeout The Prometheus shared-label tests and the auto-router session check now use it |