* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model_hub): surface model_info.description in model group info and Model Hub UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): aggregate model group description without in-place mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional
completion() only stores a custom prompt template key when its value is
truthy, and custom_prompt() already defaults both prompt values to "".
Seven call sites read them back with direct indexing, so a template that
sets only roles raises KeyError instead of rendering, on ollama, petals,
vllm, predibase, codestral, anthropic text completion, and
response_schema_prompt. Five other providers already read defensively, so
the same config works on sagemaker and 500s on ollama.
Read both keys with .get(key, ""), matching those five providers and the
defaults custom_prompt() already declares. Templates that set all three
keys render exactly as before.
* test(prompt-templates): drop the explanatory docstrings from the optional-keys regression test
* fix(health): skip background health check DB writes when the latest-row read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt-templates): read every custom prompt template field through one typed helper
Templates that omit roles, initial_prompt_value, or final_prompt_value now render with empty defaults across every provider that reads them, instead of failing with KeyError or TypeError. A field with the wrong type fails with a validation error naming it.
* fix(prompt-templates): keep templates that rendered on main working and fail bad fields with a 400
Only the roles a request uses are checked, null fields mean unset, and extra role keys are ignored, so every template main rendered renders the same. A wrong-type field now raises a BadRequestError naming it. initial_prompt_value is declared litellm-owned so it stays out of provider request bodies, like final_prompt_value. The template tests move into the mapped factory test file, and the predibase test asserts the real rendered prompt
* fix(prompt-templates): keep bad-template failover and render non-mapping role entries like main
A wrong-type template field now raises a 500 InternalServerError naming the field, so the router
still retries and cools down the bad deployment. A roles value or role entry that is not a mapping
renders unwrapped as it did on main, and role keys are only checked for roles a request uses.
* fix(prompt-templates): warn when a custom prompt template roles value is not a mapping
* fix(prompt-templates): keep deployments in rotation on a bad template and reject a template with nothing to render
* test(integration): audit custom prompt templates across endpoints, providers, and chaos
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): stop skipping rate-limit checks when token counting fails
Router._pre_call_checks() caught any token_counter() exception and
returned the full, unfiltered deployment list, bypassing the RPM,
region, and invalid-params checks for every deployment - not just the
context-window check that actually depends on the count. A content
block token_counter() cannot parse (e.g. an invalid image_url detail
value) silently disabled rate-limit enforcement for that request, with
only a log line and no error surfaced to the caller.
Now a counting failure only disables the context-window check; the
other checks still run for every deployment as before. Input-token
resolution is pulled into its own method and computed once before the
loop, so nothing needs to track "did counting fail" with a mutable
flag across iterations.
* test(router): cover the async region check when token counting fails
Replace the unit test of the private input-token resolver with a regression
test through async_get_healthy_deployments, the path the proxy uses: an
EU-pinned request whose content cannot be token-counted must not get the US
deployment
* test(router): cover the param filter when token counting fails and satisfy router coverage
* fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params
The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params,
drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every
available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away
instead of a retried 429. Deployment context windows are resolved once per call.
* fix(router): keep the retried 429 when a cooling-down deployment takes the param
* fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks
* fix(router): steer by deployment capability, apply drop settings after region and RPM checks
A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing
* test(router): make the github_copilot pre-call filter test hermetic in CI
* test(router): cover pre-call region and response_format filters through the proxy
* fix(router): judge the 400 against the request's own deployments, not a name lookup
* fix(router): judge the 400 against the passed deployments when the group name lists none
* fix(router): let a request's drop and allow settings replace the deployment's in the param check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(gemini): allow disabling thought summaries while keeping thinking on
Both reasoning-effort mappers hardcoded includeThoughts to True whenever
thinking was on, so thinking ON with summaries OFF was unreachable even though
that is the google SDK's own default. Summaries are the narration rather than
the reasoning, and for callers that discard them they are pure added latency.
Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing
changes for anyone who does not set it. It gates the includeThoughts=True
branches in the thinkingBudget mapper, the thinkingLevel mapper, and the
anthropic-style thinking param mapper.
The disable and none paths still force includeThoughts=False: with thinking
off there is nothing to summarise.
* test(gemini): assert thought-summary flag through the outbound Gemini request
* feat(gemini): let a request's thought summary ask override the gateway default
thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing
* feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default
* fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag
Normalize gemini_include_thoughts_default through normalize_drop_params so a
YAML or env value written as "false" or "0" turns thought text off and a
null or unknown value keeps the default on. thinking.display "omitted" no
longer counts as a per-request ask, so the gateway flag decides, matching
/v1/messages. Adds proxy integration cells for chat, /v1/messages and
/v1/responses that prove the flag, every per-request ask, streaming, and a
burst across workers.
* test(gemini): match the streamed Responses usage, which omits null token details
* feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks
* fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias
A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply
* fix(caching): key the response cache on the summary alias wherever the request carries it
The cache key now reads reasoning_summary and reasoningSummary through the same
reader as the summary fold, so an alias inside extra_body gets its own key too
* test: type the new summary-ask unit tests and mark their locals Final
* test: annotate the remaining summary-ask unit tests the PR touches
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: preserve non-null anyOf branches when stripping null types for Vertex
* refactor: build anyOf branches immutably and drop explanatory comments
* style: apply ruff format
* fix: annotate anyOf list construction for the LIT002 budget
* fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising
* test(vertex_ai): pin empty anyOf passthrough
* fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold
* fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing
* fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): keep user messages with None or empty list content so requests do not end on an assistant turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): avoid mutable empty-list construction in sanitizer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): avoid rebinding sanitizer input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep a trailing user turn whose content key was dropped
litellm.completion strips None fields from every message before the provider
transform, so a user message sent with content null reaches the Anthropic
converter with no content key and raised KeyError. Read the content with get
so that turn gets the same placeholder as an empty list.
* fix(anthropic): only fill user turns with empty content and keep their cache_control
* fix(anthropic): fill each empty user message in place like an empty string
* fix(anthropic): add the placeholder only for an empty turn or a cache breakpoint, in linear time
* test(anthropic): cover the tool-result guard with cache_control and widen the linear-time bound
* test(anthropic): cover empty user turns through the proxy on chat, responses and messages
* test(anthropic): type the empty user turn tests and move the linear-time check to a benchmark
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test
* fix(guardrails): record every content filter match in logging_only and offer the mode in the UI
* style(tests): format the content filter logging_only integration test
* refactor(guardrails): keep the logging_only predicate and scan override within the type gates
* fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning
* test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test
* test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects
* refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach
* test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter
* test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up
* test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): preserve provider request IDs in streaming failure logs
* fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams
Bedrock models that run through the chat-completions bridge (Claude,
converse, invoke) raise in-stream failures from the event-stream decoder
as a BedrockError without headers, so their failure logs kept
error_provider_request_id null. The decoder now passes the stream's
request-id headers to those errors.
Both the native Responses iterator and the decoder pass only the
PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs
pick up the request id and none of the 200 response's other headers.
MidStreamFallbackError gets its response at construction instead of
mutating it afterwards.
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
* fix(proxy): count endpoints return missing-key and request errors in their source API shape
* fix(proxy): drop imports the count error change left unused
* fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope
* fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures
* fix(proxy): count with the global OpenAI and Anthropic keys chat already uses
* fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway
* fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response
* fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads
* fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses
* fix(proxy): mint the OpenAI workload identity bearer inside the count error handling
* fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(router): honor explicit bedrock base model capacity
* fix(router): honor an exact base_model's capacity for every non-Azure provider
Extends the Bedrock-only check to every non-Azure provider and only uses the
base_model when it has its own exact cost-map entry, so an unmapped or wildcard
base_model falls back to the route id instead of raising in the router.
* test(router): call _maps_to_exact_model_info directly in its cost-map tests
* fix(router): let an exact base_model only widen a deployment's context window
A base_model used as a pricing stand-in with a smaller window no longer narrows the route's limit, and a route with no window never takes one from base_model
* test(router): count the base_model window helpers as covered through get_router_model_info
* fix(router): widen only from a model_info base_model naming the same model
A stand-in base_model for a different, larger model widened a smaller
deployment's window, so pre-call checks sent it prompts the provider
rejects with 400. The window now widens only when model_info.base_model
names the same provider model id as the route, which is also the field
/model/info reads.
* fix(router): match a base_model's model id across Bedrock region prefixes and @ qualifiers
* fix(router): read litellm_params.base_model too and look the base row up as written
* test(router): cover a same-model base_model window on chat, messages, responses, and complexity escalation
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The operator trace wait stopped at the root and redis spans, but the gen-AI
span is closed later by the async success callback. When it had not landed
yet, the tenant mirror was built without it and the gen-AI assertion failed.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(bedrock-passthrough): return AWS-shaped errors so AWS SDKs read the code and message
A guardrail block on a /bedrock route now answers with x-amzn-errortype ValidationException and a top-level message, and an error Bedrock returned (such as a 429 ThrottlingException) is passed through with its status, x-amzn-errortype and body
* fix(bedrock-passthrough): keep 5xx guardrail failures and failure-hook replacements unchanged
* test(bedrock-passthrough): guardrail vendor failures and unknown models keep the litellm error
* fix(bedrock-passthrough): only answer guardrail exceptions that blocked content as ValidationException
* fix(bedrock-passthrough): pass Bedrock's content-type through with its error body
* refactor(bedrock-passthrough): wrap the guardrail ProxyException instead of copying its fields
Unit tests drive the error response with the real request processor and ProxyLogging
* test(bedrock-passthrough): cover the direct model route, invoke, and auth, budget and rate-limit errors
* fix(bedrock-passthrough): pass through only errors a Bedrock model operation returned
A guardrail service's HTTP error that carries x-amzn-errortype keeps the litellm error
* refactor(bedrock-passthrough): mark Bedrock's errors where the passthrough raises them instead of matching the URL
A guardrail service's HTTP error never goes through the passthrough call, so it keeps the litellm error whatever its URL
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(cost): bill Responses API image_generation_call tool usage
Price completed image_generation_call output items from tool_usage.image_gen token counts when present, otherwise per image from the quality and size on the item, with the image model taken from tools[].model (default gpt-image-1). Add the result into tool_usage_cost alongside web search and file search, and read tool_usage from the top level or usage.tool_usage. Squashed and rebased onto main
* style(cost): annotate read-only views and mark casts for the type discipline gate
* fix(cost): price Responses image tool from tools[].model only, no default model or provider
* refactor(cost): drop redundant status check, unread usage fields and unused output_type literal
* test(cost): integration case for Responses image_generation tool token billing
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn
* test(responses): mock the websocket slot release and mark the per-turn start time naive
* style(responses): keep the per-turn start time line within ruff format
* refactor(responses): type websocket turn events as read-only mappings
* fix(responses): settle the websocket budget reservation on the first turn only
Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly
* fix(proxy): keep the legacy limiter's websocket slot until disconnect
A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache
SAML login tickets, the single-use assertion mark and cached IdP metadata
lived in user_api_key_cache, which only gets Redis when
enable_redis_auth_cache is on. With several replicas or workers the IdP
callback often landed on a process that never saw the ticket and got a
401, and IdP-initiated sign-in was refused even with Redis configured.
SAML state now has its own DualCache that _attach_redis_usage_cache
always wires to Redis, like the CLI SSO session cache, and the SAML cache
calls are async.
* test(sso): drop docstring from the SAML proxy pair fixture
* test(sso): cover the SAML login and metadata endpoints
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* test(count_tokens): expect the converted Responses input in the budgeted-key count integration test
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(guardrails): honor streaming settings on the Noma v2 guardrail
The Noma v2 initializer now passes streaming_buffer_until_moderated,
streaming_end_of_stream_only, streaming_sampling_rate and
streaming_buffer_release_on_scan from the guardrail config to
NomaV2Guardrail, so the unified streaming hook applies them. The
constructor validates them, so a custom guardrail class built on
NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults,
and an invalid value is ignored with a warning naming the guardrail, so
the guardrail still loads.
Resolves LIT-9633
* test(guardrails): cover Noma v2 streaming settings across clients, failures and workers
Adds integration cells for the custom-subclass loader, string and invalid
values, unset and false keys, a Noma outage with and without block_failures,
an upstream rejection, n=2 choices, pre_call and non-streaming guardrails,
the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a
PATCH while traffic flows, a burst while Noma is down, a slow Noma, and
replaced workers.
Resolves LIT-9633
* test(guardrails): pin the PATCH probes to one worker at a time
Each probe suspends the other worker and opens a fresh connection, so the
test checks that every worker applies the patched setting.
Resolves LIT-9633
* test(guardrails): check sampled scans by count, not window position
/v1/messages hands the streaming hook raw network reads, so where the
default sampling windows fall depends on timing; the base leg caught one
shifted window. Check the scan count and the final full-text scan instead.
Resolves LIT-9633
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
K2 in test_logging_only_scope_chaos.py flaked on CircleCI with
AttributeError: type object 'BaseModel' has no attribute '__pydantic_core_schema__'
raised inside the OpenAI client while its 30-thread control burst parsed responses
The SDKs define their response models with deferred schemas, so the first parse of each
model rebuilds it, and pydantic's model_rebuild is not safe when several threads hit it
at once. Every integration worker now builds those schemas in pytest_configure, before
any test can start a thread pool
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): preserve explicit websocket API keys
Refresh the one-line fix from BerriAI/litellm#34253 against the current default branch
Co-authored-by: puffer <389053104@qq.com>
* test(responses): verify websocket authorization
Cover explicit deployment keys, precedence over global keys, and the existing global fallback at the upstream WebSocket transport boundary
* test(responses): type the authorization capture
Replace the mutable header list with a typed, final Future carrying the upstream authorization value, addressing the review on #40853
* Refactor test_openai_websocket_authorization method
* fix(responses): resolve websocket keys with the deployment's provider
Pass the deployment custom_llm_provider to get_llm_provider so an api_base
host never swaps in another provider's env key, and drop the OpenAI-only
global key fallbacks so non-OpenAI websocket deployments resolve their own
env keys the way the HTTP route does
* fix(responses): let a deployment's own websocket key beat env keys
A deployment with an api_base on a host litellm recognizes, and no declared
provider, got that host's env key ahead of its own api_key. The explicit key
now always wins, matching the HTTP Responses route. Drop the litellm_params
api_key fallback, which could never be set because api_key is a named argument
* fix(responses): keep openai deployments on other providers' known hosts on the HTTP stream
An openai deployment whose api_base is a known OpenAI-compatible provider
host (DeepSeek, Groq, ...) opened a native Responses WebSocket there and got
a 405, because those hosts serve /responses over HTTP only. Route it through
the managed HTTP stream with the deployment's own key instead.
* fix(responses): resolve openai deployments on known provider hosts as that provider
An openai deployment whose api_base is a known provider host (DeepSeek,
Mistral, ...) now resolves to that host's provider on the Responses
WebSocket again, as before this branch, so it keeps that provider's
chat completions bridge, pricing and env key fallback. Its own api_key
still wins over the env key. The known-host test now mocks the upstream
HTTP route and checks the route, the bearer and response.completed.
* test(responses): prove websocket deployment keys and declared providers reach the provider socket
* test(responses): use the reported bare model shape for the websocket deployment key cell
---------
Co-authored-by: Ysh <734645956@qq.com>
Co-authored-by: puffer <389053104@qq.com>
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): narrow the provider response id with isinstance instead of cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): validate callback fields for current type gates
* test(gcs_pubsub): assert provider response id in spend metadata
* fix(ollama): normalize reused response ids automatically
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking
* fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds
Bedrock Invoke copied every caller thinking key when adding display updates,
so an unknown key still reached the provider with drop_params on. Filter that
copy the same way, and cover each chat call site plus the global
litellm.drop_params path in tests
* fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params
* test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions
* fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel
* feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI
Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing
early-access row, carrying the same limits and capabilities. The AI Studio row
mirrors the early-access entry; the Vertex row differs only in provider and
pricing source. Prices follow once Google publishes them.
Extend the mapped metadata test to cover all three ids, that each routes to its
provider, that its backup copy matches, and that the public rows carry the same
limits as the early-access row.
* feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops
* chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap
* fix(gemini): follow continuation hops in a loop instead of recursion
The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls
* fix(gemini): bill Google Search grounding from every continuation request
Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain
* fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI
A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream
On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued
On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports
continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch
* chore(gemini): name the type errors the dual-mode stream iterator trips
The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors
This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up
* fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens
A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request
A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate
Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk
The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment
* fix(gemini): keep one response id across a continued answer
A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream
A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key
* test(gemini): add integration cells for long decode continuation
Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs
Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once
* fix(vertex_ai): fetch the access token again before each continuation request
A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length
Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged
* fix(vertex_ai): start each continuation stream with an empty parser buffer
When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error
Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async
* fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up
A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token
Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way
* fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer
A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer
The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes
* fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata
A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex
A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not
* fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer
A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0
A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer
Adds the audit's wire tests for both, which fail 24 cells before this change
* test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index
Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed
* fix(vertex_ai): log a continued answer's raw response as one object
A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them
It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted
Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch
* test(integration): check a continued answer logs one merged raw response
A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch
* test(vertex_ai): cover a follow-up tool call and its thought signature in long decode
* test(gemini): pin that a one-request answer logs Google's body unchanged
* test(integration): assert the streamed usage.cost matches the spend row on every cost case
* test(integration): type the streamed usage frames as JsonValue
* test(integration): only require streamed usage.cost when the client received a usage frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses
* fix(ui): send detail on playground Responses image parts and code snippets
* test(messages): cover image detail on the Responses bridge wire
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it
Fixes#31449.
Two OCI GENERIC request-building defects that made Grok-on-OCI unusable
for tool-using coding-agent clients (Codex over the Responses API, Claude
Code over the Anthropic Messages API):
1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only
supports function tools")` for any tool whose type is not "function",
aborting the whole request when a client mixed a built-in/server tool
in with its function tools. Non-function tools are now skipped with a
warning; if none remain, `tools`/`toolChoice` are omitted from the
request instead of sending an empty array.
2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere
model, but OCI rejects it for all served xAI Grok models
(`Model grok-4.20 does not support parameter reasoningEffort`) and for
OpenAI's non-reasoning models (`Unrecognized request argument supplied:
reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-*
and Meta/Google models accept it. `_model_supports_reasoning_effort`
encodes that, `get_supported_openai_params` stops advertising the param
for those models, and `map_openai_params` follows the standard
unsupported-param contract (dropped under drop_params, clear error
otherwise) so `drop_params` / `additional_drop_params` now work as
documented instead of the 400 surfacing from the service.
Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model.
The two existing reasoning_effort normalisation tests move from a Grok
model to openai.gpt-5, which is the case they exercise. 21 new tests; the
OCI suite is 406 passed.
* fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools
Review follow-ups on #40732:
- `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag
of the model's `oci/` catalog entry through `get_model_info` instead of
hardcoding xAI / gpt-oss branches. Catalogued models without the flag
(xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent
from the catalog keep the historical pass-through.
- Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning
models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b`
(pricing from Oracle's public price list, 128K context per OCI docs).
- `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only.
OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output
cap silently disappears) and Llama rejects it with 400, so the catalog's
reasoning flag must not drive that switch.
- After tool adaptation, a `FUNCTION` tool_choice naming a tool that was
skipped (built-in / server tools) is dropped with a warning instead of
being sent to OCI, which rejects the unmatched choice.
Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a
forced built-in now completes and still calls the function tool; Gemini and
gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under
drop_params; gpt-5 keeps maxCompletionTokens.
* style(oci): use an immutable set for the tool-name lookup
Keeps the type-discipline gate (LIT002, mutable collections) within its
budget; the lookup is read-only.
* fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive
Review follow-ups on #40732:
- When tool_choice forces a function that is not among the adapted tools
(a skipped built-in or a misspelled name), send `toolChoice: NONE` instead
of removing the choice. Falling back to the service default would let the
model call any other supplied function even though the caller forced a
specific one, which an agent processing untrusted content must not do.
- Move the #31449 regression tests into the mapped test module
(`test_oci_chat_transformation.py`) instead of a separate file.
Verified live against OCI GenAI: Grok and Llama with a forced skipped
built-in now complete with no tool call; a forced kept function is still
honoured.
* test(oci): point reasoning_effort comments at the in-module gating tests
The referenced module was folded into this file.
* fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels
Non-function tools now follow the unsupported-param contract: a 400 naming
drop_params without it, and dropped with a warning under it, with a forced
dropped tool turning tool calls off. reasoning_effort is gated on the model's
reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the
Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop
sending a param the service rejects
* fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it
* fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name
* fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name
* fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap
* fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR
Under drop_params, non-function tools still drop when function tools remain and
tool_choice does not force a dropped tool. A request whose every tool is dropped,
or whose tool_choice forces one, now answers 400 instead of a tool-less reply the
model can fill with invented results, or a swap to another tool.
The maxTokens routing change for OCI Gemini goes back to base, since it is
outside this fix and an open PR carries it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs
* fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off
* fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off
* test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it
With modify_params on, a trailing tool search turn whose call has a kept result
got a placeholder answer from the factory and went out as a marked tool_use,
but the cache control hook counted marks before that placeholder existed and
skipped the mark, so a request at the cap reached Anthropic with 5 marks and
failed with a 400. The hook now finds client answers on the sanitized messages,
the same list the factory reads.
Also make the replay checks cheaper (prefix check before validation, early
return without provider_specific_fields, and look up client answers only when
a tool call carries cache_control), drop the two audit cells that cover old
gaps outside the tool search fix, and derive the pass-through test timeout
from the graceful stop budget.
* fix(anthropic): keep the duplicate tool result warning out of the cache control hook
The hook finds client answers with only the step of the modify_params sanitize
pass that pairs tool calls with their results, so a request with a duplicate
tool result logs the warning once again instead of once per mark count.
Also restore the Snowflake non-stream and cached-stream tool search cells as
known bugs with pytest.skip instead of deleting them, and put the Anthropic
pass-through test timeout back to 240 seconds.
* fix(anthropic): count cache marks next to a message the pairing step cannot read
With modify_params on and a marked tool call, the cache control hook paired tool
calls with their results before counting. A message without a get method (a
stray string or null in extra_body.messages, or an openai SDK
ChatCompletionMessage) made that pairing raise AttributeError before any
provider call, where main counted the marks and let the request through.
The hook now pairs only when every message can be read the way the pairing
step reads it, and otherwise counts on the messages as sent, like main did.
* fix(anthropic): count cache marks after placing a later system message the way the request builder does
* test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census
* test(anthropic): align the tool search replay cells with main and cover a system message after the search turn
The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882.
The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool
reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell
expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its
mark, and a new cell covers a system message after the trailing search turn in both stream modes
* fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request
* test(anthropic): cover marked tool calls next to a malformed tool id and a later system message
* perf(anthropic): look up client tool answers only when a marked tool call is a server tool call
Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked
tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count
marks
* test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell
* perf(anthropic): place mid-conversation system messages in linear time for models that keep them
* fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time
* test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn
* fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag
* test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router
* fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure
* test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement
* fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests
Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route.
* test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse
* fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow
* test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt
* refactor(test): avoid local list mutation and satisfy type discipline gate
* style(prompt_templates): keep line length under 120 chars for ruff format
* style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401
* style(prompt_templates): eliminate basedpyright errors in function_call_prompt
* fix(prompt_templates): keep tuple and None system content when adding the function prompt
Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged
* fix(prompt_templates): reject unreadable system content without echoing it
Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt
* fix(prompt_templates): leave non-list system content failing as before
None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated
* fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params
* test(prompt_templates): pin the function prompt on every system message
* test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): intercept the bare web_search tool type on chat completions
* fix(websearch_interception): match the bare web_search type only on chat completions
/v1/messages keeps its existing detector, so a non-Anthropic model there no
longer short-circuits a bare web_search tool into raw search results
* test(websearch_interception): cover the bare web_search type on chat and through the messages adapter
* test(integration): wait on the right signal in five flaky integration contracts
The Prometheus shared-label tests waited for the latency metric and then read the
request counter, which the logger writes after an await. They now wait for the counter
The auto-router session poll returned on the first 200 while the caller expected two
turns, so it now waits for the expected turn count
The S3 queue cap test let its failing sink recover on a 15s clock that could expire
mid-flush. The sink now stays down until a flush fails after every event is queued
K3 normally takes 168-175s against a 180s limit, so the limit is now 300s
The MCP env var browser spec reused an idle keep-alive socket that the proxy closes
after 5s, so its API calls now send Connection: close
* test(integration): add until_asserted so a wait polls the exact asserts it guards
eventually() takes a predicate separate from the asserts that follow it, which lets a
test wait on one value and then assert on another written later. until_asserted reruns
a block of asserts until they all pass and re-raises the last failure on timeout
The Prometheus shared-label tests and the auto-router session check now use it
* feat(agents): support Databricks Agent Platform agents in the agent gateway
Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire
(Responses-shaped input, output_text items, output_text.delta SSE) to Model
Serving endpoints (/invocations or the unified /serving-endpoints/responses)
and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials
Register it through the agents config block or the Admin UI agent catalog and
call it through /a2a/<agent> (message/send, message/stream) or model:
a2a/<agent> chat completions. The chat route now runs every completion-bridge
agent through its own provider, and attaches the minted Databricks OAuth token
for both bridge and URL agents. Consumed OAuth settings never reach the
provider call
* fix(agents): harden databricks_agent input, output, and OAuth handling
Non-text content parts answer 400 unless drop_params is set, a reply without
an output list answers 502, flat OAuth fields resolve an os.environ api_base,
the bridge debug log no longer echoes the request model, routing tests stub
only the HTTP boundary, and the catalog description fits the dropdown
* fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block
* fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App
* refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name
* fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty
* test(agents): drive the flat OAuth a2a send through the real bridge
* fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change
* fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents
* fix(databricks_agent): address review caveats on chat bridging, params, and auth
Map max_tokens and max_completion_tokens to max_output_tokens and pass
temperature, top_p, and user, so /v1/messages works without drop_params.
Send A2A message metadata only with string values, answering 400 or
dropping the rest under drop_params. Join message items with a blank line
in both stream and non-stream replies. Cache the env service principal
token until shortly before it expires. Keep databricks_agent replies out
of the response cache.
On the chat bridge, let request fields win over the agent's stored params
while pinning the agent's api_base and api_key, and apply the agent's
static_headers between client and agent extra_headers, as /a2a does.
Resolve os.environ/ only for databricks_agent api_base and api_key instead
of every bridge provider. Ignore an unusable databricks_oauth block on URL
agents on chat, as at the merge base, instead of answering 400.
* test(internal_context): list the databricks agent token cache as process-local
* fix(databricks_agent): keep the agent's endpoint, input, and target out of client control
extra_body model and input answer 400, or are dropped under drop_params.
A client api_base or base_url never replaces the agent's card URL on the
bridge. Chat to a URL agent keeps its merge-base call, and an incomplete
databricks_oauth block on a URL agent keeps its merge-base ValueError.
* test(databricks): expect the merge-base 400 for a client api_base on agent chat
* fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach
* fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): read a deployment's own supports_box_questions flag before the cost map
The box-question check only read the shared cost-map row and required a decisions mode there, so an OpenAI-format deployment whose model_info sets supports_box_questions was refused with a 400, which kept the wire test from #46633 red on main
* fix(decisions): keep the box flag on its deployment and out of request bodies
A model_info that is not a plain dict no longer fails the call, the proxy
decisions endpoint drops a caller-sent model_info, and the router stops
copying supports_box_questions onto the shared cost-map key.
* fix(decisions): keep the box flag shared with same-model deployments like main
Scoping supports_box_questions to one deployment turned box questions on an unflagged sibling of the same backend model from reaching the provider into a 400. Siblings inherit the flag through the shared cost-map row again, as they do on main. Also formats two new tests.
* test(decisions): pin box flag inheritance by an unflagged same-model deployment
* test(decisions): audit box-question deployment flag across providers, routes, and chaos cases
* fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body
* fix(cost): keep the type gate flat for the bridged /v1/messages billing path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
6fb24d1d read ptu_shares through the pricing terms so a termless map reserved
nothing, but with the flag off the router skips ptu_config_error for config.yaml
entries, so such an entry loaded and served every team, against the rule that a
declared split restricts access whether or not cost attribution is on.
_deployment_shares reads the declared map again, and the router now raises
ptu_config_error for a config-sourced entry declaring ptu_shares regardless of
the flag, so a malformed split is refused at boot instead of being honoured or
ignored silently.
* fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling
A name served by a wildcard route now resolves to the shared deployment behind it, so the
limiter's ceiling and the team daily activity sizing both find it. Pass-through responses
settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a
reservation taken at admission is settled even when the success metadata lost the team.
* fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard
A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids.
* test(router): pin the wildcard route precedence that PTU sizing and the router share
* fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does
A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was
refused with a 422 and dropped at load. Fractional, infinite, boolean and string
shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix,
zero-token Responses usage, fractional-count and sizing-warning paths.
* fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move
* fix(proxy): write each PTU deployment day's team charges in one transaction
The backfill counts a deployment day as priced once any team's row exists for it,
so a rollup that wrote team A's share and lost team C's left team C unbilled for
good. Each deployment day's team rows now go through one Prisma batch with the
retry around the whole batch, so a day is fully written or not written at all and
the next backfill prices it for every team.
* fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes
* test(router): list the cached PTU ceiling helpers in the router coverage check
* fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force
* fix(proxy): bill a rerun PTU day to the current share holders alone
* fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list
* fix(proxy): read the PTU flag from the secret manager first, like get_secret
* fix(router): scope retry health lookups to the caller only for PTU share holders
* fix(router): move the share-holder retry check into ptu_shares
Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them
* test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): resolve decision and judge models in caller team scope
* fix(guardrails): scope guardrail model calls to the caller's team only where it owns the deployment
Read the team only from the auth object the proxy attached to the request, never from caller JSON,
and use it only when the router's route to the guardrail model for that team reaches a deployment a
teamless caller cannot. Shared names, aliases, legacy team tags and wildcards keep resolving and
billing as before. Decision calls pass the team in litellm_metadata only when it is used, so the
logged model group stays intact.
Adds integration cells for chat, responses and messages, forged team ids, a team deployment that
shares a name with a shared model or wildcard, and billing, plus unit cases for legacy team tags
and aliases.
* fix(guardrails): route team guardrail calls to the team deployment without charging the team
Guardrail calls on a team's own model now go to the deployment the router picks for that
team, by its id, with no team on the call. Like every other guardrail call, they write no
spend row and add nothing to the team's or the key's spend
* fix(guardrails): pin the team deployment only for the judge guardrail, not shadow eval judge calls
* fix(guardrails): keep the teamless route when only a team wildcard covers the guardrail model
* refactor(guardrails): scope the team model fix to the decision model guardrail only
* fix(guardrails): send decision calls on a team's own model with the team, and keep team wildcards out
The decision call on a team's own model now carries the caller's team instead of being pinned to a
deployment id, so it is charged to that team and goes through the team's normal routing: an access
window that reserves the deployment for the team lets the call through, and pre-call checks see the
input and skip a deployment too small for it.
A team deployment listed under a wildcard public name, such as team_public_model_name openai/*, no
longer captures a guardrail model the team never named, whatever model it points at.
Adds unit and integration cells for the reserved deployment, mixed context windows under
enable_pre_call_checks, a team wildcard name over a fixed model, and the owner team's spend.
* fix(guardrails): log the owner team's name on decision calls charged to it
The decision call on a team's own model now carries the team alias beside the team id, so the
spend row and the Logs page name the team the call was charged to.
* fix(decision_model): use the caller's team only for names nothing else serves
A team model named like a shared deployment, a shared wildcard or an SDK
public name took over the guardrail's decision calls for that team, so a
team admin could point a proxy-wide guardrail at their own endpoint. The
team is now used only when a caller without a team reaches no model under
the name.
* refactor(decision_model): drop the team route subset check the served check covers
Once nothing serves the name without a team, every deployment a caller
without a team can reach is a wildcard, so the subset check only ever
rejected an empty team route. Pin team wildcards over a name nothing else
serves.
* fix(decision_model): use the caller's team only when judge_target finds nothing without a team
The served check rebuilt the router's rules and missed a shared wildcard whose
provider comes from custom_llm_provider, so a team model under the guardrail's
name took over that team's decision calls. Ask judge_target, the same lookup the
teamless dispatch makes, instead
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
