Commit graph

56573 commits

Author SHA1 Message Date
mateo-berri
99d2406a4e fix(scim): answer every SCIM route error as a SCIM Error and keep the span's error type
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Every 409 on a SCIM route now answers scimType uniqueness, so a duplicate email
that only differs in case gets it too. An unexpected error answers a SCIM Error
with the status phrase and is logged like the handlers' own errors. The OTel
server span reports the ProxyException or a public RequestValidationError the
route raised instead of ScimError
2026-11-14 10:53:47 -08:00
mateo-berri
5e7e687ebe Merge remote-tracking branch 'origin/main' into litellm_scim_error_schema
# Conflicts:
#	tests/integration/management/test_scim_user_lifecycle.py
2026-11-14 09:57:21 -08:00
mateo-berri
a63c5148a6 fix(scim): keep placeholder admin errors in the proxy shape and never drop an error's reason
The /scim/v2/placeholders list and merge routes are LiteLLM admin tools, so they
go back to the proxy's error body. An error detail SCIM cannot read now answers
its own text instead of the bare status phrase. The integration tests that read
SCIM error bodies now expect the SCIM Error schema.
2026-11-14 09:28:32 -08:00
Prince Jain
ec48e2491c
fix(aiohttp): reset the upstream connection when a stream closes early, so aborted Bedrock Invoke streams stop (#50657)
* fix(bedrock): close httpx response stream when invoke stream is aborted

* test(bedrock): use Final state holders and fixed datetime in invoke abort tests

* fix(bedrock): call response.aiter_bytes eagerly and await aclose safely

* refactor(bedrock): share one stream close helper module across routes

Move SupportsAclose, aclose_if_supported, and aiter_bytes_then_close into
litellm/llms/custom_httpx/stream_close.py so the Bedrock invoke paths, the
generic HTTP handler, and the Anthropic messages iterator use one copy, and
drop the mock-tolerant close in production code

* fix(aiohttp): drop the provider connection when a stream closes before its body ends

Closing an unfinished aiohttp response on uvloop starts a graceful TLS shutdown that keeps reading the provider's body for up to 30 s. Aborting the transport first ends the upstream read at once.

* test(bedrock): cover each invoke stream layer closing its source

* test(aiohttp): pin the TLS test certificate dates and run it through uvloop.run

* test(bedrock): assert the converse stream closes its response when an exception event ends it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 12:08:03 -05:00
mika-lt
cad4570bbe
fix(scim): skip groups with no team so a PATCH or PUT still deactivates the user (#51829)
* fix(scim): skip groups with no team so a PATCH or PUT still deactivates the user

A group the IdP pushes before LiteLLM has a team for it made the roster
write fail, which answered 404 before the user row was written, so the
user stayed active and their keys kept serving. #37700 made that failure
fatal.

Groups with no team row are now logged and left out of the roster write,
the persisted teams, and the role resolution. Group ops after a replace
now apply in order, as RFC 7644 section 3.5.2 says.

* fix(scim): drop the redundant unchanged-groups return and cover the warning and PATCH admin role

* fix(scim): strip line breaks from the teamless-group warning

* fix(scim): read the PUT role from every requested group, as POST does

A PUT naming only a group with no team no longer demotes an admin whose admin group has no team. The new tests also get typed fixtures and Final locals

* fix(scim): look up the requested teams on the primary database, as the roster write does

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 12:06:52 -05:00
norkelbyte
8140fdf54f
fix(gateway)!: apply a deployment's Messages params and fill model capabilities from the cost map (#51834)
* fix(gateway): apply a deployment's sampling params on /v1/messages

The gateway ignored temperature, top_p, top_k, max_tokens, and
stop_sequences set in a deployment's litellm_params, while the Python
proxy sends them upstream. Merge them under the client body, so a value
the client sets, null included, still wins.

* fix(gateway): apply a deployment's non-sampling litellm_params on /v1/messages

The gateway dropped thinking, tools, tool_choice, system, and the other
Anthropic Messages params a deployment sets in litellm_params, while the
Python proxy sends them upstream. Every key Python forwards now applies as a
default under the client body, so a client value, null included, still wins

A deployment's tools go first and the client's tools follow, as the Python
router merges them. Client tools that are not an array stay as sent, and
extra_body or any key outside that set still never reaches the provider

* test(gateway): cover a deployment tools value that is not a list on /v1/messages

* fix(gateway): fill model capabilities from the cost map so deployment params are shaped per model

Ports the capability fill from #50780 onto the moved litellm/rust crates, so a deployment's thinking, reasoning_effort, speed, and temperature are shaped for the model instead of reaching Claude 5.5 models untranslated and failing with a 400. Also forwards a deployment's thread param, which Python's AnthropicMessagesRequestOptionalParams lists

* fix(gateway): keep MCP server tokens out of the debug request log

A deployment's mcp_servers now reach the provider body, so the debug provider-request log would print each server's authorization_token. The logged copy replaces it with REDACTED while the provider still receives the real token

* fix(gateway): resolve os.environ/ references and honor additional_drop_params in deployment params

A deployment's untyped litellm_params now resolve os.environ/NAME at config load the way the Python proxy's config loader does, including nested objects and true/false words, so a system prompt or MCP server token kept in the environment reaches the provider resolved. A deployment's additional_drop_params now reach the Messages shaping, as on the Python proxy

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 11:51:52 -05:00
mika-lt
1f6e401a68
fix(together_ai): stop listing retired Together ids (#51827)
* fix(together_ai): stop listing retired Together ids and duplicate bare names

together_ai/* wildcard listing and get_valid_models("together_ai") returned 20 bare ids hardcoded in 2023
that Together no longer serves, next to the cost map's together_ai/ rows, so the same model could appear twice
and callers got 400 or 503 on the retired ones. together_ai_models is now built only from the cost map, like
the other cost-map-built provider sets, and validate_environment looks up a bare Together id under its
together_ai/ key so it still reports TOGETHERAI_API_KEY.

The cost map also marks DeepSeek-V3, Kimi-K2-Instruct and CodeLlama-34b-Instruct with the deprecation dates
from Together's deprecations page, so the existing deprecation filter drops them from listings

* test(together_ai): read retired Together rows through is_hidden_as_retired after main moved the helper

* fix(together_ai): use the 2026-10-30 retirement date the other Together cost map sweep uses for the three shared rows

* fix(together_ai): accept every Together key name in the key check

* fix(together_ai): keep other providers' key checks for shared bare ids and date rnj-1-instruct

* fix(together_ai): keep the constants import and stop the W&B test depending on cost map dates

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 11:15:23 -05:00
giladk1221
9d827093c5
feat(github_copilot): support GitHub App authentication with PEM upload (#51427)
* feat(github_copilot): support GitHub App installation authentication

* Add GitHub App authentication to the Copilot credential form

* Explain secure private-key setup beside Copilot App fields

* Store uploaded Copilot App keys through encrypted named credentials

* fix(github_copilot): handle App validation errors and credential replacement

* fix(ui): make GitHub App field help accessible

* fix(github_copilot): preserve saved App key references on edit

* fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health

* fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses

Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers.

App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares.

A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change.

App deployments are deployment-owned, so they share the response cache like shared device-login deployments.

App credentials are validated even when credential_info names no provider.

* fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled

The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future

* fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs

A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again

* fix(github_copilot): report a deployment naming an unloaded credential as unhealthy

The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login

* fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base

A Copilot deployment that names an unloaded credential uses the shared device login again, as on base.
A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call.
get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores.

* fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure

* fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403

Only an App deployment reports its App host as api_base before a configured api_base.
Per-user OAuth sessions report the same api_base they did on base.
The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication.
The token-dir fixture in the App api_base test is applied with usefixtures.

* fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete

* test(github_copilot): type the incomplete App settings test parameter

---------

Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 10:33:20 -05:00
paveln83
05518975a5
test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816)
* test(e2e): move audio and realtime tests off OpenAI model ids with shutdown dates

* test(e2e): move langfuse completion, image, and vertex realtime ids off dated models

* test(e2e): register the weather tool in the first realtime session update and read Langfuse v2 flattened attributes

* test(e2e): expect the OCR upload's file summary in the Langfuse generation input

The OCR upload route has sent the uploaded file, not a data URL, since the native OCR route, and the generation input summarizes that form as the mime type and file name. The broken Langfuse lookup hid the stale data URL expectation

* test(e2e): move the image generation test off gpt-image-1-mini

gpt-image-1-mini has a 2026-12-01 deprecation_date, so the image generation e2e would start failing that day. gpt-image-2.5-flare has no date and already serves the Langfuse image tests

* test(e2e): use gpt-image-2.6-flare and check every n=2 choice reaches Langfuse

OpenAI names gpt-image-2.6-flare as the successor of gpt-image-1-mini, so the image tests use it. The n=2 completions test now needs one logged message per choice, since both choices repeat the same text and a single logged message used to pass

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 09:51:53 -05:00
Michael Bennett
fd6c00200c
feat(decisions): support Vercel AI Gateway decision models (#50642)
* feat(decisions): support Vercel AI Gateway decision models

* test(decisions): cover Vercel AI Gateway dispatch through the proxy

* feat(decisions): add Vercel AI Gateway cost map rows for Jev

* fix(decisions): bill Vercel AI Gateway decisions at the charge Vercel reports

Vercel returns the charge for every call under provider_metadata.gateway.cost, already net of cache
discounts, so unpriced evaluation models and luna cache hits now bill what Vercel charged instead of
zero or the full input rate. Jev 1.14.0 and the jev alias also take box questions, as Vercel serves them

* fix(decisions): bill Vercel gateway charges with margin and BYOK, forward providerOptions

Decisions now records a provider's reported charge as upstream_usage_cost, so the
operator's cost discount and margin apply to it like any other provider-billed cost.
Vercel calls served on the caller's own provider key bill marketCost on top of the
gateway fee, the cache split comes from the upstream usage Vercel nests under
provider_metadata, providerOptions reach the gateway, remote images are inlined, and
typesafe-ai/jev, which reads text only, refuses or drops images

* fix(decisions): stop forwarding caller providerOptions to Vercel and send Jev images

Decisions no longer sends a caller's providerOptions to Vercel. Gateway
routing and provider keys are set on the Vercel team instead.

Vercel serves typesafe-ai/jev images at 768x768 (the earlier 400 came from
a 96x96 image TypeSafe rejects for its size), so the Jev rows read images
and the per-model image refusal is removed.

* fix(decisions): report provider charges on the cost header and read Vercel's cache split from the serving provider

Decisions goes back to reporting a provider's own charge through the
response cost header, the way main does for OpenRouter. Moving it onto
upstream_usage_cost dropped every cost header on free calls, priced
base_model deployments from the cost map, and put the margin in the
response header but not in the SystemOne body cost.

Vercel's cache split now comes from the provider_metadata entry of the
provider that served the call, and only when that entry reports
input_tokens_details. Any credentialType value is accepted. The Jev
1.13.0 and 1.14.0 rows record their image probes.

* test(decisions): type the async image fetch fixture in the Vercel inline image test

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 09:43:08 -05:00
norkelbyte
ac1cc5111c
fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions (#51753)
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions

* fix(proxy): check the key's max_budget on /queue/chat/completions

* fix(proxy): run budget, tag, and request policy checks on queued chat calls

* fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls

A queued stream now hands its logging object to the stream generator, so a
client that leaves mid-stream is billed for the reply it already got, and a
stream left before the first model chunk releases its budget reservation.
A team's require_trace_id and the failure logging for rejected calls now
use the same route check as the other inference checks.

* fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses

* fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls

* fix(proxy): send the call ID on a queued call the provider rejects before streaming

* fix(proxy): release a dropped stream's budget hold only when nothing bills it

A guardrail that buffers the stream can pull model chunks the client never
saw. The early-drop release settled the hold at the input cost first, then
the partial bill for those chunks added its full cost on top. The release
now runs after partial billing and only when no bill fired

* fix(proxy): fold the caller's litellm_metadata into a queued call's metadata

A queued call that sent a litellm_metadata bucket, even an empty one, made tag
routing and logging read that bucket, so the key and team tags the handler
merged into metadata no longer narrowed deployments. The queue handler now
folds the bucket into metadata and drops it, as chat does, after stripping the
proxy-owned user_api_key_* and internal control fields from both buckets, so a
caller can no longer set the queued spend row's organization ID either

* fix(proxy): release a queued call's budget hold when the call is cancelled before it answers

* fix(proxy): refund a cancelled queued call's hold in full until the model call claims it

* fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 08:54:54 -05:00
tavren27
27c796c412
fix(proxy): count the request's tool_choice on the token count routes (#51133)
* fix(proxy): count the request's tool_choice on the token count routes

/utils/token_counter, /v1/messages/count_tokens and /v1/responses/input_tokens now pass tool_choice into the local estimate, and every DeepSeek model counts it as 0, as its providers bill

* fix(token_counter): count a request with no model name again

* fix(token_counter): count grok tool_choice the way xAI bills it

xAI bills grok-4.3 and grok-4-1-fast-reasoning 21 tokens for "none", 11 for
"required" and 12 plus the name for a named tool, where the OpenAI costs
counted 1, 0 and 7 plus the name. The grok-4-1 and grok-4.3 chat template
tool sections now carry those costs, so a grok count matches its bill for
every tool_choice

* fix(token_counter): count xAI's grok tool_choice costs only where xAI serves grok

Azure AI Foundry bills grok-4-1's tool_choice differently, so grok on other hosts keeps counting it as 0. A tool_choice sent without tools counts nothing, since OpenAI and xAI reject it

* fix(token_counter): key xAI's grok tool_choice costs on the provider that serves the deployment

The count routes strip the provider prefix before counting, so Azure, Vertex AI and Perplexity grok matched the xAI name pattern. The routes now resolve the serving provider with get_llm_provider and drop xAI's costs when it is not xAI or OpenRouter's x-ai route. The costs move onto the grok tool sections so one pattern decides them

* fix(token_counter): count no tool_choice for Gemini and none without tools in the Rust counter

Gemini bills the same prompt tokens for every tool_choice, so it skips the
OpenAI choice costs the way DeepSeek does. The Rust reservation counter now
adds tool_choice tokens only when tools are present, matching the Python counter

* fix(proxy): count no tool_choice a deployment drops, and treat an api.x.ai api_base as xAI

A deployment that lists tool_choice in additional_drop_params never sends it, so the count routes no longer add its cost. An openai/ model whose api_base is on api.x.ai is served by xAI, so it gets xAI's tool_choice costs like xai/ grok

* fix(proxy): count no tool_choice a request drops in its body

/v1/messages/count_tokens takes additional_drop_params from the request body, and inference drops tool_choice when it is listed there, so the count no longer adds its cost either

* fix(proxy): take additional_drop_params on /utils/token_counter and treat a malformed api_base as not xAI

/utils/token_counter now passes a request body's additional_drop_params into the count, so a tool_choice the request drops counts as 0 there like on the other two count routes. An api_base that does not parse falls back to the deployment's provider instead of failing the count with a 500

* fix(proxy): take chatgpt and github_copilot providers from the model name in the count host check

Resolving them through get_llm_provider read the saved login, logged a warning and could refresh a token on every count with a tool_choice

* fix(proxy): skip a tool_choice or additional_drop_params too deeply nested to validate on the count routes

A value nested 300 levels deep failed TokenCountRequest validation, so the count routes answered 422 or 400 where they used to count the request without it

* fix(bedrock): count Kimi K3's forced tool_choice message on the exact count path

Kimi K3's chat template adds a tool-choice system message when Bedrock gets toolChoice any or a named tool, and Bedrock bills it (170 input tokens unforced, 206 forced). The exact count path now passes tool_choice through the same Converse mapping inference uses and renders that message

* test(proxy): type the Kimi K3 count test's tools parameter

* perf(count_tokens): validate each kept count option once instead of twice

* fix(dot_notation): parse drop paths in linear time when brackets never close

* test(count_tokens): benchmark the unclosed-bracket drop path instead of timing it in a unit test

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 07:53:27 -05:00
KVR Vishnu
8b28818fa2
fix(openrouter): suppress empty post-finish choice before usage trailer (#50210)
* fix(openrouter): drop empty post-finish choice from usage trailer

OpenRouter's accounting chunk after finish_reason repeats an empty
assistant delta (role + finish) alongside usage. After usage was
retained for cost tracking, that leftover choice was still emitted
between the terminal finish and the final usage chunk (#50156). Clear
choices on post-finish usage trailers so clients only see content,
finish, then usage.

Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com>

* test(openrouter): cover sync and no-stream_options post-finish usage trailers

* test(openrouter): type the post-finish trailer test and cover async over a sync stream

* fix(streaming): skip the usage trailer in async iteration of a sync stream when usage is off

The async-over-sync loop returned a usage chunk unchanged when the caller did
not ask for usage, so the post-finish trailer arrived as a chunk with no
choices. It now records the usage and skips that chunk, as the native async
loop and the sync loop already do.

* fix(streaming): keep a post-finish choice that carries a payload when dropping the empty trailer choice

The usage trailer copy cleared every choice, so images, audio, thinking
blocks or logprobs that a provider put on that chunk never reached the
caller. It now drops only a choice whose delta holds nothing beyond its role
and empty content and that has no logprobs.

* test(streaming): read the payload test's streamed choices through one comprehension clause

* fix(streaming): set the final chunk's hidden usage when async iteration reads a sync stream

* style(streaming): drop the comment that restates the post-finish usage branch

* fix(streaming): keep the final chunk's hidden usage when include_usage is false

Dropping the empty post-finish choice also dropped the chunk that carried OpenRouter's usage to callers who set include_usage false, so stream_chunk_builder fell back to a token guess with no cost. The final chunk's hidden usage and its post-loop correction now apply whenever no visible usage chunk is sent.

* refactor(streaming): bind the async-over-sync returned chunk with a walrus

---------

Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 07:48:43 -05:00
quietjules
007d1c1b5e
feat(proxy): support budget_limits windows on internal users (#51763)
* feat(proxy): support internal user budget windows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): make user budget migration repeatable and sync API types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): bound lock wait for user budget migration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): omit unset JSON windows when creating users

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): retain personal budget windows for proxy admins

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): give user budget windows their own counters and keep them on the user

User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path

* fix(proxy): keep windows on team auto keys and write all users' windows in one transaction

* fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock

An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets.

JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do

* fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock

Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses

The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits

* test(proxy): check default window resets against the reset helper instead of the wall clock

* fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes

* fix(proxy): fail an all-users bulk update whole when its transaction fails

The per-user fallback after a failed all-users transaction wrote the newest
user and then crashed on a missing user_email attribute, so the call answered
500 with one user changed. Racing window bulk updates past the 5s transaction
limit reached it. Answer 500 with no user changed instead

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 07:44:35 -05:00
norkelbyte
1f2b2734e5
fix(logging): send one async success log per sync bridged Responses stream (#51787)
* fix(logging): send one async success log per sync bridged Responses stream

A sync litellm.completion(stream=True) on an openai/responses/ model called
each CustomLogger's async_log_success_event three or four times for one call.
The outer CustomStreamWrapper runs a per-chunk async_success_handler on the
logging thread pool while the inner Responses iterator logs the completed
response on the caller's thread. A per-chunk run that passed the early-return
check before the completed response was stored re-read the shared
model_call_details at callback time, found the key the other run had stored,
and logged a second success instead of a stream event.

Each run now decides once whether it delivers the final stream log, from
whether it stored the assembled (or pass-through) response itself, and every
stream branch of the delivery loop (CustomLogger, ShadowEvalLogger, custom
functions, openmeter, dynamodb) uses that decision

* fix(logging): gate v2 stream delivery on the run's own final log

Chunk logs on a sync stream run on executor threads, so the shared
async_complete_streaming_response key can appear while a chunk run is
between its early return and the v2 gate. include_v2 now uses the same
per-run delivers_final_stream_log flag as the legacy delivery.

Also types the race test callbacks.

* fix(logging): gate sync string-callback stream logs on the run's own final log

The sync success handler still read the shared complete_streaming_response key to decide whether supabase, logfire, lunary, helicone, s3 and athina get the final stream log. A sync stream chunk run that read the key after the final run filled it sent those integrations the finished response again, so one call wrote the same s3 log 3 to 6 times. Each sync run now uses its own assembled response, like the async handler

* fix(logging): send sync stream callback functions and chunk-logging callbacks only the final log

A sync stream ran success callback functions and the promptlayer, wandb, athina, and traceloop callbacks once per chunk log, so a cost tracking callback summed one call's cost several times. They now skip a run whose result is one streamed chunk, which keeps the single log of a sync file content stream

* fix(logging): decide the sync stream chunk gate before message redaction

With turn_off_message_logging on, the sync handler redacted a stream chunk to a dict before checking whether the run carried a chunk, so callback functions and the promptlayer, wandb, athina, and traceloop callbacks still logged every chunk. The run now records that before redaction

* fix(logging): send a sync stream's end log once even when its response builder fails

A sync stream whose stream_chunk_builder raised ended with a best-effort usage chunk, which the chunk gate skipped, so success callback functions and the promptlayer, wandb, athina and traceloop callbacks got no call. The stream wrapper now sends its end log once through stream_end_success_handler, which the gate lets through, and skips the second submit after an include_usage chunk

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 07:25:03 -05:00
mika-lt
d81c55f4cc
feat(custom_llm): let custom providers serve native generateContent (#51083)
* feat: support native generateContent in CustomLLM

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve custom generateContent accounting and built-in priority

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: forward custom generateContent success callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve custom generateContent failure callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: retain async callbacks across generateContent fallback

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(google_genai): scope custom generateContent hooks to native fields and stream only with a stream hook

* fix(google_genai): log custom adapter calls on the caller's logging object and reject untranslatable config

Custom adapter calls now share the caller's logging object, so the proxy's deferred logging covers them and one request logs one row under the caller's call id and call type. Native hook responses log from any pydantic, SDK, or dict shape, per-request prices apply to custom handlers, and a missing stream hook or a generationConfig key the adapter cannot send answers 400

* fix(google_genai): return custom hook objects as their camelCase dict so the router tags the deployment

* fix(google_genai): keep tool args, drop SDK-only fields, and let routers retry custom legs that cannot stream

Tool call args and function responses keep the caller's keys and null values. SDK objects no longer send sdkHttpResponse, automaticFunctionCallingHistory or parsed to the client or the logs. A custom leg without a stream hook answers 500 like CustomLLM's other unimplemented hooks, so a mixed router group retries the request on another leg. The wire-value walk is depth capped for the recursive function check

* fix(google_genai): keep router legs serving chat after a custom stream without a hook, and accept config the adapter already meets

* fix(google_genai): reject JSON mode a custom leg's completion adapter cannot send

The custom-leg generationConfig check now accepts responseMimeType and response schema keys only when the adapter sends them as response_format, or when they ask for plain text. JSON mode without an object schema answers 400 unless drop_params is set

* fix(google_genai): send a custom hook's dict back without SDK-only fields

A hook that returns a dumped google-genai response as a dict sent its snake_case keys, parsed, sdk_http_response and automatic_function_calling_history to the client, past the guardrail that rewrites only candidate text. Dicts now go back as the same filtered camelCase body as every other hook return, and the router tags that copy instead of writing into the hook's own dict

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 05:57:58 -05:00
tavren27
6fc368a15c
fix(azure): send no api-version on Foundry project /openai/v1 Responses bases (#51767)
* fix(azure): send no api-version on Foundry project /openai/v1 Responses bases

Azure answers 400 "api-version query parameter is not allowed when using /v1 path" for every api-version on a Foundry project's /api/projects/<p>/openai/v1/responses route, so drop it there and keep it everywhere else

* fix(azure): send no api-version on Foundry project v1 chat and WebSocket, bill routed models

Chat completions on a Foundry project /openai/v1 base now use the v1 OpenAI client, so they send no
api-version, which that path rejects. The Responses WebSocket URL builder strips a bare /openai/v1 base
before adding /openai/v1/responses, so it no longer dials a doubled path. An azure/model-router call that
routes to a model priced only under azure_ai/ is billed from that row plus the router fee once

* fix(azure): keep the Azure client off chat-only paths, honor api_version none, drop api-version from v1 base queries

Only chat completions switch to the v1 OpenAI client on a Foundry project /openai/v1 base, so transcription and
azure_text keep the Azure client and no longer fail inside LiteLLM. A deployment with api_version none sends no
api-version on Responses and chat, for gateways that hide the project path. The v1 client now moves an api_base
query into default_query without api-version instead of gluing the route after it

* fix(azure): pick the v1 client for o-series chat, keep api_version none off a doubled /openai/responses path

* fix(azure): send no api-version on embeddings under api_version none, test async and streaming chat on v1 bases

* fix(azure): join openai/v1 once onto a v1 client base that ends in /openai

Chat and embeddings under api_version none, and chat under api_version v1, built /openai/openai/v1 from an api_base ending in /openai, while Responses on the same base already built /openai/v1. The v1 client now joins the route the same overlap-aware way

* fix(azure): keep a percent-encoded gateway path as typed on the v1 client base

Join openai/v1 by string instead of add_path_to_api_base, which decodes %2F, %25, %3F and %23 in the base path. Type the chat test helper's mode as a Literal with assert_never.

* test(azure): import assert_never from typing_extensions so the Azure tests load on Python 3.10

typing.assert_never needs Python 3.11, and the package supports 3.10

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 05:24:12 -05:00
paveln83
3ad164382b
fix(responses): send cache_control as a prompt_cache_breakpoint on the chat-to-Responses bridge (#51783)
* fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them

A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to
prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a
function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode
caching never wrote. The bridge now renames the marks before converting the messages when the
served deployment reads breakpoints, using the same gate the Responses config uses for native
/v1/responses input. Deployments that do not read breakpoints keep today's behavior.

The rename also leaves unmarked content parts untouched instead of validating them, so a part with
non-string keys no longer raises on either path.

* fix(responses): keep leading system messages in client order once one goes to input

A marked first system string became a list and moved to input while the
next unmarked string stayed in instructions, so the model read them in
reverse. The leading run that joins instructions now stops at the first
system message whose content is not a string

* fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge

Bedrock Mantle already caches the growing prefix on its own, so turning
cache_control into breakpoints there added write charges. The client-order
rule for leading system messages now applies only when a mark turns a system
string into a list, so unmarked chats send the same request as before.

* test(responses): cover a marked user turn in the unmarked system split test

A cache_control mark on the user message must not switch unmarked leading
system messages to the client-order rule. The split test now runs with the
user turn plain and marked, so dropping the role check from the system
string test fails it.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 05:00:49 -05:00
paveln83
7d4c0ab91a
fix(proxy): retrieve and delete a managed file when the request also names a model (#51780)
* fix(proxy): retrieve and delete a managed file when the request also names a model

GET and DELETE /v1/files/{id} sent a proxy-managed file id to the provider unchanged whenever the x-litellm-model header or ?model= named a model, so the provider answered 404 and DELETE left the file in place. A managed file id now skips model-based routing and resolves to each deployment's provider file id, as the content route already did.

* fix(proxy): keep forwarding a managed file id by the model hint on a proxy without a database

A proxy with no database loads no managed files hook, so it cannot resolve a
managed file id itself and must keep sending it to the named model, which may be
another LiteLLM proxy that owns the file. Skip model routing for a managed id only
when the managed files hook is loaded.

The tests now run the real managed files hook over a stored row with only the
OpenAI file routes faked, and cover the no-database proxy.

* test(proxy): type the managed file table helper's delete filter

* test(proxy): keep a raw provider file id routed by the model hint when managed files are on

A proxy with a database loads the managed files hook, which now skips model routing for its own file ids. This test makes sure a raw provider file id sent with a model hint on that proxy still goes to the hinted deployment with its credentials, so the skip cannot widen to every id unnoticed

* test(proxy): cover a model hint that names a model the managed file is not on

A hint naming another deployment must still resolve the managed id through the managed files hook, so these cases fail if routing is skipped only when the hint matches the file's own model

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:55:36 -05:00
tavren27
4e29aea11c
fix(responses): accept a string or tuple allowed_openai_params on Responses models instead of answering 500 (#51751)
* fix(responses): accept any sequence for allowed_openai_params and give UnsupportedParamsError a response

A deployment whose allowed_openai_params is a string (or an SDK caller passing a tuple) made every
Responses-mode request answer 500 "can only concatenate list". The Responses path now builds the
allowlist from any sequence, matching the chat path.

UnsupportedParamsError now runs BadRequestError's constructor, so it carries response, body and
request like every other 400 instead of raising AttributeError on .response.

* fix(params): read a bare-string allowed_openai_params as one param name on every path

A deployment or request with allowed_openai_params: "presence_penalty" now allowlists presence_penalty
on chat completions, embeddings, the Responses API, the chat-to-Responses bridge, the router's tier
and precall filters, and Mistral's prompt_cache_key check. Before, chat and Responses read the string
letter by letter, so it allowlisted nothing, and the router ignored it.

UnsupportedParamsError's args now hold its own message instead of the BadRequestError one.

* fix(params): read a dict or comma-separated allowed_openai_params the way chat and embeddings did

A dict allowlist is read for its keys and a string is split on commas, so configs that worked on
chat or embeddings before keep working, and the router reads them the same way. The fallback errors
header names an error by its class when its type is empty, so UnsupportedParamsError still shows up
there by name.

* fix(params): read every param name in a string allowed_openai_params

A string allowlist now yields every name in it, whether the names are split by commas or spaces or
written as a list inside the string, so embeddings configs like "dimensions encoding_format" keep
working. A request with no allowlist returns early instead of raising and catching a validation error.

* fix(fallbacks): keep the fallback errors header type for errors other than UnsupportedParamsError

The header named an error by its class whenever its type was empty, which changed "None" to the class name for
timeouts and other errors built without a provider body. Only UnsupportedParamsError needs its class name now that
it inherits an empty type, so every other error keeps the type it carries.

* perf(params): read a string allowed_openai_params with one regex scan

The string reader walked every character in Python and ran twice per request, so a 1 MB allowlist added about
300 ms of event-loop time. `\w+` matches exactly the alphanumeric-or-underscore runs the old loop kept, checked
over every Unicode code point, and runs in C.

* test(params): cover an int allowed_openai_params and an admin request for a model two teams share

An allowlist that is no string, mapping, or sequence now allowlists nothing instead of raising TypeError, and an admin request with no team for a model name two teams share answers the multiple-teams 400 with the model name

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:32:38 -05:00
zhengqiuyang
ad24ca9f31
fix(cohere): honor COHERE_API_BASE for embeddings (#50249)
* fix(cohere): honor COHERE_API_BASE for embeddings

The cohere embedding branch in the dispatch passed the caller's api_base
straight through with no env fallback, so COHERE_API_BASE was silently
ignored for embeddings — the request went to the hardcoded
api.cohere.ai/v2/embed host. The rerank path (rerank_api/main.py) already
reads the same variable, and the databricks/hosted_vllm/openrouter
embedding branches follow the same convention; this branch was the
asymmetric one (same family as voyage's #49411).

Adds two respx regression tests: the env-routed gateway URL is used when
the variable is set, and the hardcoded default still applies when it is
not.

Signed-off-by: 郑润泽 <506166760@qq.com>

* fix(cohere): route COHERE_API_BASE through ensure_api_route and pin the url semantics

Review follow-up on #50249: COHERE_API_BASE means different things across
cohere's own paths (chat reads it as the full /v2/chat endpoint, rerank as
a base URL) and the PR initially used the verbatim reading — which breaks
a bare-host gateway silently (the request would land on the gateway root).

get_complete_url now runs whichever value it receives through
ensure_api_route(..., "v2/embed"): a bare host joins the route, a full
/v2/embed endpoint passes verbatim, and the no-env default is unchanged.
The default domain (api.cohere.ai vs the api.cohere.com the siblings use)
is left alone on purpose — flagging it here for a maintainer call instead
of bundling a behavior change into this fix.

Tests: three get_complete_url unit pins (bare host joins, full endpoint
verbatim, default unchanged) plus the env-routed respx test reshaped to
the bare-host form. 76 passed in tests/unit/llms/cohere/embed/.

Signed-off-by: 郑润泽 <506166760@qq.com>

* fix(cohere): fail fast when COHERE_API_BASE is a chat endpoint; pin verbatim-path semantics

Review of #50249 surfaced two gaps:

1. Chat reads COHERE_API_BASE as a complete URL (the value may end in
   /v2/chat or /v1/chat), while embeddings now treat it as a base to join
   /v2/embed onto. A chat-configured deployment upgrading would get
   embeddings at <chat-url>/v2/embed and 404. get_complete_url now fails
   fast with an actionable error when the value is a chat endpoint, so
   the user configures embeddings separately instead of silently losing
   them on upgrade.

2. The added tests only covered a bare gateway host — the one shape
   where join and verbatim agree. Adds a /v1/embed verbatim pin (the
   issue's original shape) and a chat-endpoint rejection test.

The query-string corruption in ensure_api_route (pre-existing, locked by
test_url_utils.py) is real but out of scope here — it predates this PR
and needs its own change plus a decision on the locked assertion.

Release-note-worthy: chat-only deployments that set COHERE_API_BASE and
then configure embeddings will now see the fail-fast error instead of a
silent 404.

Signed-off-by: 郑润泽 <506166760@qq.com>

* fix(cohere): resolve the embed api base without rebinding the parameter

AGENTS.md asks for no variable mutation and no comments that restate the code. The base is now a Final local next to cohere_key, and the three-line comment is gone. py_compile passes; the local pytest run is blocked by an unreleased native symbol (NativeRedisClient), so CI verifies the suite.

* fix(cohere): keep an explicit embedding endpoint verbatim

The route was appended to any path that did not already end in /v2/embed, so a gateway exposing /v1/embed or /embeddings received /v1/embed/v2/embed. An explicit embedding endpoint is now used as given, and a chat endpoint still fails fast.

* fix(cohere): keep the configured URL out of the chat-endpoint error

The message reaches proxy error responses, and a server-configured URL can carry credentials in its userinfo or query string.

* fix(cohere): raise BadRequestError for a chat-endpoint base and fix the verbatim test

Review on #50249: aembedding wrapped the guard ValueError into APIConnectionError, so the proxy answered a network error instead of 400. Raise litellm.BadRequestError and assert it. The explicit-endpoint test compared against api_base.rstrip(/) while the code returns the base untouched; compare against api_base. Also drop a corrupted character in the message.

Signed-off-by: 郑润泽 <506166760@qq.com>

* fix(cohere): send embeddings for a chat-endpoint COHERE_API_BASE to the same host's embed route

* fix(cohere): keep an explicit embed api_base as is and derive COHERE_API_BASE safely

An explicit api_base is the full request URL again, as before this PR.
Only COHERE_API_BASE is turned into an embed URL: a trailing /vN,
/vN/chat or /vN/rerank is dropped, /v2/embed is joined on the path so a
query string survives, and only a path ending in /embed is used as is.
litellm.api_base is no longer consulted for Cohere embeddings.

* test(cohere): drop an extra blank line in the embed URL tests

* fix(cohere): read COHERE_API_BASE for embeddings from the environment only

A proxy with a secret manager no longer makes an extra lookup on every
Cohere embedding call, and a bare /chat or /rerank path is now covered
by tests.

---------

Signed-off-by: 郑润泽 <506166760@qq.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:30:11 -05:00
Mateo Wang
1a8ed27237
refactor: clear fresh debt in customer hierarchy and v2 callback modules (#51804)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 01:14:06 -08:00
jesus-berri
3988ce2865
fix(proxy): honor cancel_on_disconnect on the rerank endpoints (#51252)
Some checks are pending
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / mcp-elicitation (push) Blocked by required conditions
unit / mcp-server (push) Blocked by required conditions
unit / proxy-auth (push) Blocked by required conditions
unit / proxy-endpoints (push) Blocked by required conditions
unit / proxy-extras (push) Blocked by required conditions
unit / proxy-feature-endpoints (push) Blocked by required conditions
unit / proxy-hooks-client (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(proxy): honor cancel_on_disconnect on the rerank endpoints

* test(proxy): cover a connected rerank succeeding with cancel_on_disconnect on

* test(proxy): cover a connected rerank answer through the cancel watcher in the integration rig

* fix(proxy): read cancel_on_disconnect on rerank the same way the chat path does

* fix(proxy): set up the rerank request logger before routing so a hang-up after dispatch logs the query's tokens like chat

* fix(proxy): keep flag-off rerank logging unchanged and price a cut-off rerank from its query and documents

The rerank endpoint now builds its request logger only under cancel_on_disconnect, so a flag-off rerank logs one unbilled 408 row as on main.

The failure estimate for rerank counts the query and every document sent, prices the deployment the router picked, and bills one query on a model priced per query.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:59:22 -05:00
Yogeshwaran Ravichandran
35b9df1f40
feat(mistral): add native upstream token counting support (#50652) (#50653)
* feat(mistral): add native upstream token counting support

* test(mistral): standardize on mistral-large-4 and add tool calls tests

* fix(mistral): unify credential resolution and guard Azure endpoint boundary

* fix(mistral): resolve api base from provider env vars and route responses count natively

* fix(test): use groq for bridged proxy tests and increase mistral coverage

Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov.

* fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter

* fix(mistral): suppress basedpyright warnings on untyped httpx handler

* fix(mistral): keep chat credential resolution and count with the key chat resolves

The token counter now resolves its Mistral key and base through the same helper chat uses, so
a count goes to the endpoint chat would call. A deployment that would route to Azure, or an
Azure-only env key, falls back to the local estimate instead of sending that key to Mistral.
Chat key and base precedence is unchanged from main.

* fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route

* fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them

Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge,
which restores previous_response_id history. The count now takes the same bridge instead of forwarding
previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone.

The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an
input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them.

Tests now go through the real handler with respx, and the integration Mistral peer serves the count route.

* fix(mistral): leave empty assistant turns out of the count, as Mistral chat does

Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count
endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400
through previous_response_id, so the count now skips the same turns chat skips.

* fix(mistral): count what chat sends and keep the bundled Mistral model list

A user turn that mixed file_id files with an image or an inline file lost the image
or inline file in the count, since the converted content arrives as a tuple and only
lists were read. Both content readers now take either shape

A /v1/responses count with a json_schema text.format now sends that format to
Mistral, so the count matches what the bridged chat call bills

Counts for Mistral models that are not chat models (embeddings, OCR, transcription)
now fall back to the local estimate instead of returning Mistral's 400

Count calls now use the deployment's timeout when one is set, and a count response
without input_tokens falls back to the local estimate

MistralConfig.get_models goes back to the bundled model list, so turning on
check_provider_endpoint does not change what /v1/models shows for mistral/*

The previous_response_id count test now fakes the spend logs table instead of the
history lookup, so the real lookup and its query run

* fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral

* fix(mistral): answer 408 on a count timeout and use the shared deployment timeout

Main's native counters answer a count timeout with 408 and a connection
error with 500. The Mistral counter fell back to a local estimate on both,
so a hung Mistral endpoint looked like a successful count. It now raises the
same count errors, and it reads the deployment timeout with the shared
deployment_timeout helper instead of its own copy

* fix(mistral): join count text blocks the way Mistral chat does

* fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat

* fix(mistral): count the thinking chunks Mistral chat replays in assistant history

The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content
carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is
skipped the way chat skips it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:19:57 -05:00
quietjules
9ba216d521
fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239)
* fix(vertex_ai): reuse one Google token across vector store searches

* fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst

* test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy

* test(vertex_ai): type the fixtures and token callback in the concurrent search test

* fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects

A shared RAG Engine config made one store's slow token fetch hold up every
other store, and kept a rotated key file's deleted key until a restart.
First loads and expired refreshes now take a lock per credentials and project,
refreshes take a lock per credentials object, and any RefreshError reloads the
credentials once.

* fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls

* test(vertex_ai): check async callers sharing an expired credential refresh it once

* fix(vertex_ai): reload Google credentials only when a refresh is rejected for good

A retryable refresh error such as a 503 now raises without reloading the key, so an outage
does not double google-auth's retries. Credentials that cannot be weak-referenced share the
config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase
and holds the token response until the other callers have started

* fix(vertex_ai): reload Google credentials only when Google answers invalid_grant

google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads

The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did

* fix(vertex_ai): pick up a swapped Google key file on the next call

The credential cache key now carries the key file's inode, mtime and size, for a
vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced
or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of
after Google rejects the old key or the proxy restarts

* fix(vertex_ai): keep the earlier key while a replaced key file does not load

A key file caught halfway through a non-atomic write, or replaced by one that does not load,
now keeps the credentials loaded from its earlier version in use and logs a warning, as the
cache did before key files were re-read. The next complete file is loaded on the next call.
The pub/sub test no longer writes a key file it only reaches through the seeded cache

* fix(vertex_ai): try a replaced key file again when its first token fetch fails

The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one

* fix(vertex): raise when an async reload keeps an earlier key Google rejected

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:12:09 -05:00
tavren27
c4c16c7b3f
fix(azure): drop the traceback from stream-cut errors sent to clients and alerts (#51760)
* fix(azure): drop the traceback from stream-cut errors sent to clients and alerts

An Azure chat or text completion stream cut mid-response raised an APIConnectionError whose message carried the full Python traceback, so the in-stream error frame, the /v1/completions 500 body, and every llm_exceptions alert showed server file paths. The Azure no-status branch now keeps the error line alone, and route_failure_proxy_exception redacts the message it answers clients with, the way the chat route already does

* fix(proxy): keep route error messages as raised and test the Azure stream cut through the proxy routes

Drop the route-level client message redaction, which rewrote valid guardrail and hook 400 messages, and cover the Azure chat and text stream cut end to end through respx

* fix(azure): name the error type when a connection error has no message

An httpx reset raises ReadError with an empty message, which left the client line with no cause once the traceback was gone

* test(proxy): drop the docstring from the Azure stream cut fixture

* test(proxy): stop pinning the MidStreamFallbackError prefix in the Azure text stream cut test

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 08:00:58 +00:00
quietjules
93f8d86c89
fix(gateway): return Claude thinking blocks on chat completions instead of a 500 (#51774)
* fix(gateway): return Claude thinking blocks on chat completions instead of a 500

The gateway's Anthropic chat transform answered 500 on any thinking or redacted_thinking block. It now maps them into reasoning_content and thinking_blocks the way the Python proxy does.

* fix(rust): replay Claude thinking blocks a chat client sends back on its next turn

A client that sends a thinking reply back now gets it replayed to Anthropic the way Python does it, dropping blocks Anthropic could not have signed and unwrapping Bedrock-wrapped redacted data, where the gateway used to decline with 501

* fix(rust): replay a cut-off Claude thinking reply and send a repeated thinking block once

A reply that runs out of max_tokens mid-thought comes back with content null and thinking_blocks. Sending it back now replays its thinking blocks alone, the way Python does, instead of answering 501

A thinking block that several merged assistant messages carry goes to Anthropic once, where it first appears, like Python's _without_repeated_thinking

New and changed tests use rstest, as litellm/rust/AGENTS.md asks

* perf(rust): drop repeated thinking blocks in one pass

* fix(rust): send a thinking reply on to OpenAI-compatible models like Python

Python passes chat messages to OpenAI-compatible endpoints verbatim, so a Claude reply carrying thinking_blocks and reasoning_content reaches the model as sent. The gateway declined it with 501 instead

* fix(rust): drop null message fields before an OpenAI-compatible call like Python

Python's cleanup_none_field_in_message removes None fields before the call, since some endpoints reject them

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:44:23 -05:00
paveln83
dcba0cf6f0
fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content (#51747)
* fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content

Foundry's Fireworks-hosted Kimi K3 takes reasoning_effort as a level string, but azure_ai refused it (400 on /v1/messages with thinking) or dropped it. Models whose cost-map row declares reasoning_effort_levels now get the param, rounded up to a declared level, with deployment-declared levels taking precedence. A dict effort sends its level, and one without a level answers 400 or drops under drop_params. FW- models also keep reasoning_content on replayed assistant turns, which Kimi reads back across tool calls

* fix(azure_ai): read declared reasoning levels from the loaded catalog

The router copies a deployment's model_info, reasoning_effort_levels included, onto the shared azure_ai/<model> key in litellm.model_cost, so reading that live map let one deployment's levels open reasoning_effort for an undeclared sibling. Reading the catalog as loaded keeps the gate on what the cost map itself declares, while deployment levels still round the request through the deployment path

* fix(azure_ai): pass Foundry reasoning levels through and match the catalog row regardless of case

Foundry Kimi K3 accepts every reasoning_effort level, so the level is sent as the caller wrote it instead of rounded onto the catalog row. "default" leaves the level to the model. A value outside the level names answers 400, or is dropped with a warning under drop_params. An effort object sends its level and warns that its other fields are dropped. The catalog row lookup now also matches a key that differs only in case

* fix(azure_ai): read base_model for Foundry reasoning effort and replay, accept auto

* test(azure_ai): cover global drop_params for a Foundry effort that is not a level

* fix(azure_ai): keep replayed reasoning_content for Foundry kimi- models

Foundry's MoonshotAI-format Kimi deployments (kimi-k2.6, Kimi-K3.1-Pro) read
replayed reasoning_content back, so it is no longer stripped for them. The
effort-object warning now names only the fields it drops.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:08:11 -05:00
tavren27
d76cdcbb8a
fix(proxy): list key and team aliases on /model_group/info and apply listing callbacks to them (#51754)
* fix(proxy): list key and team model aliases on /model_group/info

* fix(proxy): keep listing-callback and pause filters on model alias rows

A name a legacy or V2 listing callback hides stays hidden on /v1/models,
/v1/models/{model_id} and /model_group/info even when it is also an alias.
V2 Hook.list_models subscribers now limit which target an alias row
describes, /model_group/info describes no alias by a paused model, and
?model_group=<name> offers listing callbacks only the names that row reads.

* test(proxy): annotate the new alias test locals as Final

* fix(proxy): keep a caller alias listed when a router alias of the same name points at a hidden model

A key or team alias always wins over a router model_group_alias of the same name, so alias rows are now dropped only for names a listing callback hid itself, not for router aliases whose target it hid

* fix(proxy): describe an alias by an unlisted target only when listing callbacks keep it

Listing callbacks are now also offered the unlisted model each listed key or team alias
sends requests to, and the model a router alias of that name resolves to. The alias row
is described by that model only when the callback keeps every one of those names, and a
name the callback hides there hides no listed name. Before, any installed listing
callback, V2 included, left such an alias with no limits and a 404 lookup

* fix(proxy): describe an alias to a team model's public name when the key lists its internal name

/model_group/info lists team models by their internal routing name, so an alias to the
public name found no listed target and got no row while /v1/models listed it. Alias
resolution now reads the listed names by their public names. The listing callback
docstring also names litellm.model_alias_map among the alias sources.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:10:32 -05:00
zoveki
8973e2dc4f
feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project (#50732)
* feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project

Adds the opt-in litellm.vertex_request_label_fields setting, the Vertex twin of bedrock_request_metadata_fields. When set, Gemini, embeddings, Imagen and Gemini image generation, rerank, and Claude on Vertex (chat and /v1/messages) send the listed proxy identity fields and spend_logs_metadata as request labels, identity first, with the user_api_key_ prefix reserved so a caller cannot forge or evict them. Identity text is rewritten to fit Google's label rules. Gemini image generation now forwards labels at all.

* feat(vertex_ai): send the opt-in identity labels on multimodal embeddings, the Google GenAI route, and batches

With litellm_settings.vertex_request_label_fields on, the multimodal embedding
predict body, the native generateContent and streamGenerateContent body, and the
BatchPredictionJob now carry the key's identity labels the same way chat does

Caller labels on GenAI are resolved after the extra_body merge, so neither a
top-level labels field nor extra_body.labels can write the reserved prefix. Batch
labels come only from the proxy's litellm_metadata, never the caller's OpenAI
batch metadata. With the setting off every route sends the same body as before

* fix(vertex_ai): keep identity labels when a Gemini caller sends extra_body labels

* fix(vertex_ai): label Interactions, image edits, Mistral, and the /vertex_ai pass-through, and gate Gemini image labels on the setting

* fix(vertex_ai): move body labels into the labels header on Vertex rawPredict pass-through calls

* fix(vertex_ai): send Mistral extra_body labels in the labels header instead of the body

Labels a caller nests in extra_body now merge over top-level labels before the
X-Vertex-AI-Labels header is built, the way the Gemini body merges them, and
leave the outbound body. The token-fetch stubs in the Mistral and multimodal
embedding tests are scoped to their instance so later tests see the real method

* fix(vertex_ai): limit pass-through labels to POST and to publishers that take the labels header

rawPredict to custom endpoints and publishers other than Anthropic and Mistral, and PATCH or PUT bodies, now reach Vertex as the caller sent them. The pass-through drops a caller's labels header where it writes body labels, a single field written as text turns the setting on for that field, and an undecodable Claude labels header falls back to the caller's labels

* fix(vertex): label Veo video, Gemini transcription, and pass-through predictLongRunning requests

* fix(vertex): read the pass-through publisher from the end of the upstream path

A project id named publishers shifted the first-match lookup, so a forged X-Vertex-AI-Labels header reached Claude or Mistral untouched

* fix(vertex): drop a caller labels header on body-labelled Vertex routes while identity labels are on

* test(vertex): script the v1beta1 GenAI target in the labels header integration test

* test(vertex): type the transcription test's access token override

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 00:39:01 -05:00
Mateo Wang
c6aa2a3b2a
test(logging): scope bridged call recorder to its request (#51339)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:34:56 -08:00
Mateo Wang
e87d7bd714
fix(anthropic): restore sync messages streams for OpenAI and Azure (#51305)
* fix(anthropic): return a sync stream from sync messages.create on openai and azure models

* fix(anthropic): keep the sync messages stream readable with async for and end it on close

* fix(responses): close the chat completions stream when a sync messages stream closes

* fix(anthropic): release sync streams on async close

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:34:21 -08:00
Mateo Wang
ccb7f753c9
fix(mistral): send Magistral reasoning as supported native levels (#51329)
* fix(mistral): send reasoning_effort and thinking to magistral models as a level Mistral accepts

magistral-small-latest and magistral-medium-latest now take reasoning_effort
(none or high) as their reasoning switch, but LiteLLM dropped reasoning_effort
and thinking for any magistral model and injected Magistral's deprecated
reasoning system prompt instead, so streamed calls returned no reasoning and
"none" still reasoned.

Forward reasoning_effort rounded to the cost map's declared levels, map
thinking onto it (disabled becomes none), drop the system prompt, and declare
none and high on both magistral rows.

* fix(mistral): send adaptive thinking to magistral as a reasoning level

* fix(mistral): reject reasoning_effort strings that are not a level, and leave "default" to Mistral

A string such as "HIGH" or "auto" went to Mistral as sent and Mistral answered 422, even with drop_params. It now takes the same path as a number: dropped with a warning under drop_params, a 400 without it. "default", which litellm.completion lists as a value, sends no reasoning_effort so Mistral applies its own default.

* fix(mistral): preserve thinking when unsupported effort is dropped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(mistral): raise reasoning failures at the public boundary

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mistral): share fixed capabilities with warning regressions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:33:54 -08:00
norkelbyte
8decbe4801
fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources (#50577)
* fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources

An s3:// URL in an image_url or file part now becomes a Converse
s3Location source instead of failing with "Unsupported image type".
The format comes from the part's format or the key's extension, an
optional bucket_owner is sent as bucketOwner, and an unknown or
unsupported format answers 400. An image in an assistant turn no
longer crashes the async path.

* fix(proxy): limit non-admin s3:// media to allowed buckets and refuse managed prefixes

A non-admin key may only send an s3:// media URL whose bucket is listed in
general_settings.allowed_client_side_s3_buckets (or when client-side
credentials are allowed), and never one under litellm's managed prefixes.
Proxy admins are not limited. A bucket_owner on an s3:// URL that is not a
non-empty string is a 400, the Responses bridge forwards the owner only for
s3:// URLs, audio on S3 is a 400, and video in a tool result is sent to Bedrock.

* fix(proxy): check s3:// media in every request field and refuse it in Bedrock batch files

The s3:// media check now reads the whole request body instead of a fixed
list of fields, and it also reads Gemini file_uri and fileUri, so a
/v1beta generateContent request with an s3:// file part gets the same
bucket rules as chat, Responses, and Anthropic requests. A Bedrock batch
input line with an s3:// media URL is a 400 before upload, since Bedrock
batch files do not take s3:// media URLs yet.

* fix(bedrock): say that LiteLLM refuses s3:// media in Bedrock batch files

The 400 for a Bedrock batch input line with an s3:// media URL now says
"LiteLLM does not accept s3:// media URLs in Bedrock batch files", since
Bedrock batch takes S3 locations and it is LiteLLM that refuses the line.

* fix(proxy): check s3:// URLs only inside media parts and refuse them only on Converse batch lines

The s3:// bucket check now reads URL fields only inside the content parts that a Bedrock transform turns into an S3 source, and it uses the same s3:// test as the prompt factory. Values in metadata, annotations, tool arguments and other fields pass as before. Each URL is named once in the 403 and 400 messages

Bedrock batch files refuse s3:// media only on the lines that go through the Converse transform, so passthrough-provider lines upload unchanged

* test(proxy): check that a media part's non-URI fields never count as s3 media

* fix(proxy): check s3 URIs inside encoded file ids and on queued chat completions

The s3 media check now decodes model-embedded and unified file ids, since
completion() turns them back into the raw s3 URI they carry. The
/queue/chat/completions endpoint skips common request processing, so it now
runs the same check. The media walk goes level by level instead of
recursing, which the recursive_detector code-quality step requires.

* fix(proxy): let a key send managed file ids it may read and refuse s3 sources hidden in model_file_id_mapping

The s3 media gate refused every LiteLLM managed file id whose upload sits under the
managed prefix, even when the key's own user or team created it. It now asks the
managed files hook which unified ids the key may read and skips only the objects
those ids embed. A forged or unowned id, or a proxy without the hook, still gets a 403.

The gate also reads the provider file ids in a client-supplied model_file_id_mapping,
so an alias can no longer point a non-admin request at an unlisted or managed object.

Markdown documents (.md, .markdown) now map to Bedrock's md document format, so an
s3 Markdown source without an explicit format no longer fails with an unknown media type.

* fix(proxy): check s3 media on every Responses WebSocket frame and keep inline tool-result video dropped

Every response.create frame on WS /v1/responses now runs the same s3 media check as
HTTP, so a non-admin key can no longer send an unlisted or managed s3 object over a
WebSocket

A tool result forwards a video to Bedrock only when it is an s3Location. An inline
base64 video is dropped with a warning, as it was before this PR, since Converse
rejects video in a toolResult block

A non-admin key that sends an AWS identity its deployment admits through
configurable_clientside_auth_params (aws_role_name, aws_web_identity_token or
aws_profile_name) skips the bucket list, because Bedrock then reads the object with
that identity. The managed-prefix 403 still applies. A WebSocket frame's own AWS
identity fields never open the bucket list, since the managed WebSocket handler
does not forward them and Bedrock reads with the deployment's identity

The check now skips the body walk when no string starts with s3:// and no file_id or
model_file_id_mapping key is present, which keeps large image bodies cheap

* fix(proxy): let a client AWS identity skip the s3 bucket list only when Bedrock signs with it

An admitted aws_role_name, aws_web_identity_token or aws_profile_name now skips the
bucket list only when it decides the credentials get_credentials picks, on every
deployment of the model group. A web identity token without a role and a session
name does not, since get_credentials then ignores it. A profile does not when the
deployment or the environment sets a role, since the role branch runs first. In
those cases Bedrock still reads with the proxy's credentials, so the bucket list
applies

BaseAWSLLM.identity_choosing_params mirrors the branch order of get_credentials, and
both read the same AWS_* environment fallbacks through one helper

* fix(proxy): keep the s3 bucket list for non-admin keys that send a client AWS identity

A client AWS identity field does not prove that the caller can read the bucket. A
role or profile name picks credentials the proxy holds, and Bedrock reads the object
with them. So the bucket list now applies to every non-admin s3 media part, whatever
AWS identity fields the request sends. allow_client_side_credentials: true still
turns it off

This removes BaseAWSLLM.identity_choosing_params and the router lookup that only the
skip needed

* fix(proxy): look up managed file ids only when they can change the s3 refusal

* fix(bedrock): check every file id field, skip MCP tool calls, accept registered S3 media types

* fix(proxy): refuse unlisted s3 media on the rag query and adapter routes

* style(proxy): format the merged collections.abc import

* fix(proxy): walk tuples when checking s3 media and skip plain number lists

* fix(proxy): skip numbers when walking a request body for s3 media

A body with large token-id lists and any s3:// string or file_id key made
the full walk build a node per number, taking seconds per request. The walk
and its prefilter now skip int, float, bool and None values.

* test(proxy): check the s3 refusal through the real request setup

* test(proxy): exercise real setup for allowed s3 media

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(model-armor): fail closed on remote s3 documents

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): suppress cyclic fallback payload inference

Limit the suppression to reportGeneralTypeIssues on the legacy payload assignment. The fallback payload is constructed as a dict and the runtime AST is unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:33:21 -08:00
Nate Armstrong
90122c07a2
ci(integration): point the rust gateway install step at litellm/packaging (#51743) 2026-11-14 13:36:57 +09:00
mika-lt
7ab37a1529
fix(count_tokens): answer 400, not 500, for a message without role or content on Anthropic and Azure AI (#51284)
* fix(count_tokens): answer 400, not 500, for a message without role or content

Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.

* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count

* refactor(bedrock): leave message validation to the Bedrock count handlers

Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy

* fix(count_tokens): name the chat message chat translation cannot send

A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back

* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content

Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it

* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse

The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate

* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse

A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again

* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given

* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks

* test(integration): expect the 400 for a count whose leading system message has no content

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:21:29 +00:00
norkelbyte
dc378de207
fix(cli): let Claude Code renew a --pkce login's key through lite's apiKeyHelper (#51319)
* fix(cli): let Claude Code renew a --pkce login's key through lite's apiKeyHelper

* fix(cli): keep an explicit --api-key over lite's apiKeyHelper and persist its absolute path

An explicit --api-key or LITELLM_PROXY_API_KEY is now exported to Claude Code
even when lite's renewing helper is configured, since the helper prints the
stored login's key instead. The helper command also uses the absolute path of
the lite found on PATH, so a relative PATH entry cannot make Claude Code run
another repository's bin/lite

* fix(cli): keep an explicit key and --pkce when lite's renewing helper is set

A --settings that blanks apiKeyHelper now keeps the exported key, and a key passed with --api-key turns lite's helper off for that launch, so Claude Code no longer warns about two credentials or sends the stored login's key beside it

The helper names the lite installed beside the Python that ran the login before one found on PATH, so a pyenv shim no longer fails in projects pinned to another Python

lite claude signs in again with --pkce when the stored login used it, and print-token's hints name the proxy and --pkce

* fix(cli): keep a shell token from beating the renewing helper and reuse --pkce only for the same proxy

The --pkce --config-claude login now writes an empty env.ANTHROPIC_AUTH_TOKEN beside the helper, since a token exported in the shell wins over apiKeyHelper

lite claude sends any other key through a private settings file that blanks the helper and merges the user's own --settings

A re-login keeps --pkce only when the stored login is for the same proxy, else the saved default decides

* fix(cli): let any lite install's helper renew for its proxy and name the renewing login in print-token hints

lite claude now treats a lite auth print-token helper for this proxy as renewing whichever lite path it names, since every lite reads the same token store

Every print-token hint to sign in again, including an expired key and an unreadable keychain, names lite login --pkce with the proxy

Shorter docstrings

* fix(cli): send a launch key to the proxy it was issued for

The private settings file lite claude writes for a key that lite's helper would not renew now also sets ANTHROPIC_BASE_URL. Without it, lite --base-url B --api-key <B key> claude after a --pkce --config-claude login to A sent the B key to A, since the login's settings point Claude Code at A

* fix(proxy): preserve caller metadata on auth failures (#51745)

* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs

Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.

* fix(proxy): type the spend logs metadata header parser honestly

The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.

* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket

A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.

* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler

Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.

* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer

The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.

* test(proxy): read auth failure spend rows from a real unconnected PrismaClient

Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>

* fix(logging): keep a content-block system prompt in logged messages (#51699)

* fix(logging): keep a content-block system prompt in logged messages

A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.

Fixes #36402

* fix(logging): type the logged system message

* fix(logging): type the system message without a mutable annotation, close the test client

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter (#50483)

* test(mcp): check the tool dict conversion without patching the MCP handler

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter

* refactor(mcp): type the semantic filter's default tool loader

Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.

* fix(mcp): give the MCP client exactly the servers left on in the connect page (#50973)

* fix(cli): send the launch key under a project apiKeyHelper and give each key its own settings file

A `lite login --pkce --config-claude` leaves an empty ANTHROPIC_AUTH_TOKEN in Claude Code's user settings. When a repo's project settings set their own apiKeyHelper, `lite claude` skipped its private settings file, so the empty token hid the exported key and Claude Code sent the project helper's key. `lite claude` now also writes that file whenever the settings files leave ANTHROPIC_AUTH_TOKEN empty

Every key launch also wrote the one file `--subscription` uses, so two launches started together could read each other's key and proxy. A key launch's file is now named after a hash of its contents, and every overlay is written to a temp file and renamed so a starting Claude Code never reads half of one

* fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)

* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

* test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cli): skip the working directory when the login looks up lite for its helper

On Windows, shutil.which searches the current directory before the directories it is given, so a lite.exe planted in the repo where someone ran `lite login --pkce --config-claude` could become their user-level apiKeyHelper

The login now checks each directory by full path, this Python's scripts directory first and then each PATH entry, skipping empty entries

* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)

* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved

Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.

* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows

* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs

* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params

* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params

* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase

* fix(bedrock): mark the service tier block field read-only

* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params

* test(prometheus): sort the service tier label test imports

* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400

* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova

* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk

The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one

* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name

The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it

* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added

The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

* fix(cli): delete the key files lite claude leaves on logout and unconfigure

lite claude hands its process to Claude Code, so the --settings files holding
its key stayed in the Claude config dir. lite logout and lite unconfigure claude
now delete them, and Claude Code reads one only at startup, so open sessions
keep working

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: joshua-berri <joshua@berri.ai>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: shrey <shrey@berri.ai>
Co-authored-by: Ankit Jha <45779665+PyExtreme@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: quietjules <349048734+quietjules@users.noreply.github.com>
2026-11-14 04:04:38 +00:00
zoveki
01de1a686d
feat(opensearch): map vector store search filters to k-NN filters and add hybrid search (#51131)
* feat(opensearch): map vector store search filters to k-NN filters and add hybrid search

* test(code-quality): allow the bounded OpenSearch filter recursion

* fix(opensearch): retry an unfilterable engine without filters under drop_params, match strings on keyword subfields, keep per-request field keys

Under drop_params, a 400 'does not support filters' from an nmslib or older
engine re-runs the search once without filters, as filtered searches behaved
before. String eq, ne, in and nin match the key's .keyword subfield, falling
back to the key where no subfield is mapped. Only the index name, search
pipeline and hybrid flag are registration-only, so opensearch_text_field and
opensearch_embedding_field pass through per request again

* fix(opensearch): match string filters on the exact keyword field and embed once on the filter retry

String filters now read the index mapping and compare the exact keyword field, refusing a key the index cannot match exactly. The drop_params retry reuses the first embedding and also covers OpenSearch before 2.4. An index that refuses filters without drop_params answers litellm's own 400, and a filters value that is not an object stays a 400 under drop_params. The Admin UI store form gains the search pipeline and hybrid search fields

* test(ui): expect the OpenSearch search pipeline and hybrid search fields in the provider field list

* fix(opensearch): drop string filters under drop_params when the mapping read fails, and skip normalized keyword fields

* fix(opensearch): name the permissions a refused index mapping read needs instead of returning OpenSearch's body

* fix(opensearch): check the index mapping for every filter, resolve aliases, and keep failed mapping reads from dropping filters

* fix(opensearch): keep an alias filter's own name and refuse rewrites that reach an index without the alias

* fix(vector_stores): sign the caller's headers as given and reject filters on keys only some OpenSearch indexes map

A null or non-str extra_headers value no longer fails Bedrock Knowledge Base, S3 Vectors, or other provider searches before signing. The mapping read gets only the string-valued headers.

A filter on a key that some of a store's indexes map and others do not answers 400 (drops under drop_params), since OpenSearch returns its k-NN hits on some calls and none on others.

* fix(opensearch): send the caller's headers on the index mapping read as given and name the fix for keys only some indexes map

A header httpx cannot send now fails the filtered search on the mapping read, the same way it fails the search, before the query is embedded. Bytes headers now reach the mapping read too.

* test(opensearch): pin that a timeout on the index mapping read answers 408 without embedding the query

* fix(opensearch): answer 400 for a filter key inside a nested field or naming an object

A root-level term query on a field under a nested mapping, or on an object field, matches nothing, so the search answered 200 with no hits. These keys now answer 400 naming the mapping, or drop the filters under drop_params

* fix(opensearch): filter on a nested field whose values OpenSearch copies to the root

A nested field mapped with include_in_root, or include_in_parent on it and
every nested field above it, has its values in the root document, so a root
term query matches it. Those filters now search instead of answering 400.
A key that names an object or nested field answers that it holds fields
rather than values before the nested check, so its remedy no longer leads
into a second 400

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:30:55 +00:00
moe-berri
752eb7c5b7
fix(opencode): stream Go Messages-protocol models through chat completions (#51715)
* test(opencode): cover Go Messages chat streaming

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): delegate Go Messages chat streaming to Anthropic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): preserve Messages stream usage and cleanup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 22:21:52 -05:00
Hoya
8ef451a187
fix: raise on failed OpenRouter non-streaming completions (#50880)
* fix: propagate OpenRouter errors and preserve failure usage

* test: cover OpenRouter errors and failure accounting

* fix: bill a recovered OpenRouter failure on the success log and parse the body once

A retry or fallback that recovers from an OpenRouter finish_reason error now
carries the failed attempt's usage and cost into the success log, so the key
pays for both calls. The response body is parsed once and handed to the error
check, and the failure-log test no longer waits on a 10-second deadline

* fix: carry a failed attempt's usage before each SDK retry

* test: cover OpenRouter's documented failure body without usage

Pins the recovered-failure test's start time to a fixed date and adds a regression test for the documented non-streaming error body, which carries no usage

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:02:00 +00:00
Nate Armstrong
cd2e4b6b60
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0)

* feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope

* fix(custom-logger-v2): add field projections and keep the request scope across copies

* fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line

* fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists

_safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones.

* fix(custom-logger-v2): contain a failing v2 callback on logging hooks

A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged.

* fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines

run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path.

* fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point

Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes.

* fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope

The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected.

* fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object

Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged.

* refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests

Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow.

* fix(custom_logger_v2): await a single-entry v2 stage in the caller's context

A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather.

* fix(custom-logger-v2): raise a native callback panic as a runtime error

* fix(custom-logger-v2): run a synchronous v2 line on a private event loop

* fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler

* fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure

* fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope

* fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once

* fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure

* fix(callbacks): reject a selection name or alias that is not lowercase

* fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks

* fix(router): type the fallback event kwargs handed to the v2 dispatch

* feat(custom_logger_v2): type the contract fields as records and mark derived-only fields

Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy

* feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts

A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers

* feat(callbacks-python): hand the logging payload to native callbacks by reference

NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy

* feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity

Typed members keep the bool, int, float or str python holds through a Scalar
type on both sides, so ids, limits and priorities no longer lose values that
lax pydantic refused. A member the contract cannot carry is named in rejected
so a port can reproduce what python does with it.

Logging hooks get logged_key_identity, built from the payload metadata the
python loggers read on every route, while key_identity stays on request hooks
and keeps reading the auth object.

* fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways

budget_reservation read the dict budget_reservation.py builds with a KeyError
on callback_bound. Undeclared keys are now ignored, callback_bound is a member,
entries keep a non-mapping item as null so capacity's model access group check
stays exact, and costs read as the float budgets computes with float(x or 0.0).

Float members cross as the raw json marker when they are not finite, and a
native verdict returns through json_to_py, so an infinite or NaN cost survives
a read and a write instead of turning into null.

* feat(callbacks): carry a python value as its str() text and truthiness in scalar members

Scalar gains a Stringified variant with the str() text and bool() of a value
that is not a bool, int within i128, float or str, so a port can reproduce
the label, path or log text python formats and the falsiness of an empty
container without confusing it with a real string. Budget reservations say
whether python held the entries as a list.

* feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary

A Scalar member now carries a list, dict, set, bytes, object or an int past
i128 as its str() text and truthiness instead of naming it in rejected, which
stays for values whose str() or bool() raises. request_tags carries any
iterable as the items prometheus iterates, and budget reservation entries
carry what budgets iterates with entries_is_list for capacity's list check,
never written back to the shared dict.

* feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int

A Stringified scalar now carries hashable, from trying hash() at the
boundary, so a port can raise the TypeError capacity's priority lookup and
the limiter's descriptor identities raise for a list, dict or set and not
for a tuple or object. It also carries integer, so an int past i128 is told
apart from any other value whose text is digits.

* fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does

Prometheus treats any guardrail_mode that is not a list as a single mode, so a
tuple mode holds no str mode and a missing mode is one None mode. The
derivation unpacked tuples and rejected other non-str modes instead.

* feat(callbacks): let a native handler await typed host calls the python caller serves

* test(callbacks-python): drive a native handler's host calls through the python dispatcher

* test(custom_logger_v2): route a native logger with a host through the execution driver

* fix(custom_logger_v2): type the wrapped guardrail modes for pyright

* feat(custom_logger_v2): carry the python type name and hash error text of a stringified value

* feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it

* feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none

* feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs

* feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one

* fix(callbacks-python): render null keys like a python dict merge, adding a missing key last

* feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar

* feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload

* feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end

* feat(custom_logger_v2): carry the start and end times prometheus reads from the call details

* feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call

* fix(callbacks): export the limiter request record

* test(custom_logger_v2): give the type name cases stable ids and sort the record exports

* fix(custom_logger_v2): type the new derivations and tests for pyright

* fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads

* style(callbacks): rustfmt the record exports

* refactor(custom_logger_v2): rename native callbacks to rust callbacks

native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle,
NativeState and HostedNative renamed to their rust forms. The pyo3 classes
NativeCallback and NativeRequestState become RustCallback and
RustRequestState, and the tests, fixtures and messages follow.

* refactor(custom_logger_v2): rename slots to field sources

slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is
SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A
table that only derives is named DERIVED_*_FIELDS, and a table that can
write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The
include_v2 comment moves down to the functions that take it.

* refactor(litellm_logging): place the v2 members of Logging beside what they serve

v2_scope, v2_auth and v2_failure_delivered join the other per-request
class defaults, v2_logging_scope and its opt-out helper follow
should_run_callback, v2_call_facts sits by the standard logging payload
builder, and claim_v2_failure sits above failure_handler.
bind_v2_request_scope checks for a Logging object directly instead of two
runtime protocols.

* refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate

* refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field

A logged payload that is present but not a mapping now looks the same to a
rust callback as a missing one, a parity gap accepted for a case that does
not occur. metadata_labels, input_sequence_length and limiter_request move
down to bits 44-46, on both sides of the contract.

* feat(callbacks): type a rust callback's request state

Callback gains an associated State type: request_state returns it and
handle and handle_hosted receive Option<&State> beside the input, which no
longer carries the state. ErasedCallback is the object-safe form the python
wrapper stores; it carries the state type-erased and downcasts it once,
returning CallbackError::StateType rather than panicking on a mismatch.
handle_hosted's default names its unused host _host. The record sample
count follows the dropped logged_payload_not_a_mapping field.

* docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are

* refactor(callbacks-python): render payloads with python-compat's json writers

pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr.
python-compat's json module now exposes write_string, generic over a String
or byte sink, and float_text, and the payload renderer uses them.

* docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet

* docs(custom_logger_v2): add a README for the package

* fix(custom_logger_v2): import the field sources tests from field_sources

* test(callbacks-python): stub the litellm package in each payload test

* fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check
2026-11-14 02:33:27 +00:00
norkelbyte
5849906ddb
fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079)
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route

/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.

DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.

* perf(proxy): delete count drop paths from one copy of the options

additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear

* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing

Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills

Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400

additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500

The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models

* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's

A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference

A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does

The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any

* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude

A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status

/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none

* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking

A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts

A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it

* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking

A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count

* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle

* fix(bedrock): send the thinking beta headers on Mantle counts

A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends

* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test

* fix(deepseek): count only the tools DeepSeek chat inference sends

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:29:16 +00:00
mika-lt
b77a1295a5
fix(gemini): stream TTS audio in delta.audio on chat completions (#51750)
* fix(gemini): stream TTS audio in delta.audio on chat completions

* fix(logging): drop audio from logged responses when message logging is off

turn_off_message_logging blanked the text of a logged response object
but left message.audio and delta.audio in place, so callbacks reading
response_obj or the assembled stream still got the whole clip. The dict
path already cleared audio. Streamed Gemini TTS now carries audio, which
made this reachable on a second path.

Also pin the TTS stream test to gemini-3.8-flash-tts.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:27:55 +00:00
quietjules
fa79311dfc
feat(model-catalog): read min_billed_input_tokens (#51128)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:16:53 +00:00
quietjules
f5ef14308b
fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved

Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.

* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows

* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs

* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params

* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params

* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase

* fix(bedrock): mark the service tier block field read-only

* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params

* test(prometheus): sort the service tier label test imports

* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400

* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova

* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk

The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one

* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name

The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it

* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added

The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:33:57 +00:00
devin-ai-integration[bot]
03f14051b9
test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 01:05:05 +00:00
Ankit Jha
5f381ad472
fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)
* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:03:19 +00:00
shrey
68d6db45ef
fix(mcp): give the MCP client exactly the servers left on in the connect page (#50973) 2026-11-13 16:36:49 -08:00
Mateo Wang
c4a72b8fac
refactor(mcp): let the semantic filter take its MCP tool loader as a parameter (#50483)
* test(mcp): check the tool dict conversion without patching the MCP handler

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter

* refactor(mcp): type the semantic filter's default tool loader

Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.
2026-11-13 16:34:00 -08:00
ryan-crabbe-berri
0f97850c50
fix(logging): keep a content-block system prompt in logged messages (#51699)
* fix(logging): keep a content-block system prompt in logged messages

A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.

Fixes #36402

* fix(logging): type the logged system message

* fix(logging): type the system message without a mutable annotation, close the test client
2026-11-13 16:23:29 -08:00