The sync failure tests swapped litellm's own Langfuse and Lunary loggers for MagicMocks. The Lunary test now runs the real LunaryLogger with the SDK's track_event recorded, and the Langfuse test moves next to the Langfuse export fixture, where a real LangFuseLogger exports to a recorded OTLP route.
tests/unit/AGENTS.md forbids unit tests that depend on execution speed, so the four large errors now run as a benchmark that asserts the scrubbed output.
The scrub walked every "..." in the error and tried each quote around it to find
the two ends pydantic shows of a long input. A request of dots made that walk cost
about 48 Python steps per character on the event loop, with no cap tripping.
Replace everything from a line's first input_value= to its last , input_type=
(or to the line's end without one) instead. Finding both labels is linear, the
scrub no longer needs the prompt's cut sides, and the cut-side work cap goes with
them. A value that is not request text, such as None or a bad role, is now blanked
too. A quote of the prompt that holds input_value= is replaced first, so the label
replace can't split it. Other quotes go after, and a text's trailing label is left
out of its quote, so a message such as "value" or ", input_value=" can't break the
label of a real error.
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows
Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts.
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03
A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503
* fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again
Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(azure): strip output-only reasoning content from Responses input
* test(azure): cover idless encrypted reasoning replay
* fix(azure): preserve existing status handling during reasoning replay
* fix(azure): scope reasoning content filtering by provider
* fix(azure): drop status and content from replayed reasoning items
* fix(azure): forward non-object input items and drop reasoning content on every Azure route
* fix(azure): drop replayed reasoning content only under drop_params
Reasoning status is still dropped on every Azure route. Caller-set reasoning
content now follows the created_by convention from #48783: it is dropped when
drop_params is on and forwarded otherwise, so Azure answers its own 400.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Every string the Models + Endpoints, Model Insights and Transform Request
pages write themselves now goes through t() or Trans, including the shared
auto-router settings those pages render on other routes. 1,651 English
messages are added under 28 new top-level keys in en.json.
This is a pure text move. The English on screen does not change, option
lists and constants keep their shape, and text that could not move without
restructuring stays English. ModelInfoEditForm.tsx, ComplexityRouterConfig.tsx
and build_complexity_router_config.ts grew past the 800 line limit and get
max-lines entries in eslint-suppressions.json instead of being split.
Move batch B1 of the admin dashboard onto react-i18next: the dashboard
shell (navbar, command palette, deployment health, header, layouts), the
entry pages (login, onboarding, change password, home, API reference,
UI theme, workflows), the public AI hub and the AI hub table, the shared
components and helpers in src/components/shared, common_components,
networking toasts, the toast titles and the shadcn primitive labels.
Pure text move into t() and Trans with 1,056 new en.json keys. The
English on screen is unchanged.
* fix(bedrock): add GovCloud Kimi K3 runtime prices
* fix(bedrock): omit retired max_tokens from Kimi rows
* fix(bedrock): add GovCloud Haiku and Grok prices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): match GovCloud Grok 4.7 limits to the card and date GovCloud Claude 3 Haiku
* fix(bedrock): add priced Grok Mantle base for regional routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass answered overload status to failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve Anthropic route exception handling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): bind the transformed failure without rebinding the parameter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe the overload failure hook through a real ProxyLogging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(least-busy): reserve the picked deployment so a burst of requests spreads out
The in-flight count only went up in log_pre_api_call, which runs after
selection and the pre-call hooks. Requests selected in that window all saw
the same counts and landed on the same deployment (16 simultaneous
requests over 4 deployments went as lopsided as 12/4/0/0 in the proxy).
When the router selects for a request it is about to send, least-busy now
records a reservation for the chosen deployment in the same synchronous
step that reads the reservations, and log_pre_api_call turns it into the
normal in-flight count. Selections without request kwargs (token counting,
probes) do not reserve. A reservation whose request never reaches the API
call expires after 30 s. The shared and per-worker counts and their
success/failure decrements are unchanged.
* fix(least-busy): prune expired and empty selection reservations
The reservation map kept a key per model group forever, so a wildcard
route with caller-chosen model names grew it without bound. Drop expired
and empty entries on every write.
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
* fix(least-busy): keep pick reservations in the shared router cache
Selectors that share a router cache dedupe to one registered callback, so
only that instance saw log_pre_api_call. Reservations held on the other
instances were never released and skewed picks for the full TTL. Storing
them in the router cache's in-memory layer lets any selector release them,
and the per-key TTL replaces the full-map rebuild on every write.
Tests now hold the burst with a barrier instead of a sleep and use a manual
clock object.
* fix(least-busy): restart the reservation TTL on every write
InMemoryCache keeps the TTL a live key was first given, so a reservation
key that kept being rewritten still expired 30 s after its first pick and
took the newer reservations with it. Drop the key before each write so the
TTL counts from the latest one, and leave it deleted once it is empty.
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
* fix(router): count least-busy decisions calls and release reservations on cache hits
Least-busy now reads the model group from litellm_metadata before metadata, so /v1/decisions calls that carry caller metadata are counted in flight and release their reservation at pre-call. A response cache hit releases its reservation instead of decrementing another request's in-flight count, and Responses WebSocket turn admission selects without reserving. The reservation store is reached through a typed protocol.
* fix(decisions): count proxy /v1/decisions calls in least-busy's in-flight totals
* fix(least-busy): keep reservations out of the shared router cache and count requests that start during a Redis read
Least-busy now reads a call's deployment from the routing metadata the router sets, so /v1/decisions logging params stay as on main and failed decisions calls no longer start cooldowns.
* style(router): wrap the least-busy reservation store line
* fix(least-busy): lock the reservation read-and-update so threaded sync calls keep each other's picks
Router.completion from a thread pool and sync success callbacks can read and rewrite a deployment's reservations at the same time, so one thread's pick or release could be lost.
* fix(least-busy): release a pick whose call ends before reaching the provider
A call that fails or returns before the provider handoff (a 400 for an unsupported param, token counting, a proxy-gate rejection without a start log, batch create) now drops its reservation instead of decrementing a count it never added. Router.select picks and batch cancel no longer reserve, since neither makes a call that logs.
* test(least-busy): give callback fixtures a fixed call-start time
The routing-group tests fed least-busy a finished call without the start time litellm stamps before counting it, so the count stayed up after the fix. Fixtures now use a fixed timestamp instead of the clock.
* fix(least-busy): free tier-group picks, stop lookups reserving, and count only started attempts
Reservations are keyed by deployment id, so a complexity, auto, adaptive or
quality router frees its tier pick when the call starts. Each started attempt
is recorded under its call id, and only a recorded start lowers the count when
the attempt ends, so a fallback hop rejected before the provider frees its pick.
Whether a pick reserves now comes from a context variable set by the routed
call sites, so get_available_deployment and the deprecated
async_get_available_deployment no longer hold a deployment, and custom routing
strategies and Router subclasses keep their method signatures.
* fix(least-busy): free a request's unstarted picks when it ends
Each pick now carries its call id, so a start frees only its own call's pick, and the router frees every pick a request never started once that request ends. A fallback hop rejected before the provider no longer leaves its pick reserved for 30 seconds.
* fix(router): free a least-busy pick when a vector store create ends
* fix(router): free least-busy picks of calls made without a call id
* fix(router): free the least-busy pick of a sync text completion
* test(router): use fixed start times in least-busy fixtures
* fix(router): free the least-busy pick of a sync generic call that fails before its provider
* fix(router): read in-flight counts and start attempts under one lock so a busy deployment never looks free
* fix(router): keep pass-through lookups from reserving least-busy picks
* test(least-busy): wait for the helper thread to start before the timed join
* fix(least-busy): keep realtime sessions out of in-flight counts
---------
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill Batch cached tokens at the Standard cache price when a row has no Batch cache price
* fix(cost): resolve the Standard cache price for the prompt length in the Batch fallback
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GitHub only issues Copilot tokens through copilot_internal/v2/token to its own
Copilot apps, so a custom GITHUB_COPILOT_CLIENT_ID signed users in and then
failed with no Copilot access. With a custom client ID, look up the account at
copilot_internal/user for the seat and Copilot API address and send the user's
GitHub token straight to Copilot. The default client ID keeps the token swap.
* ci(unit): fetch the merge base by sha for the db-and-spend schema check instead of cloning full history
* ci(unit): read the schema check arguments through one typed mapping
* ci(unit): give the schema check its own step with the ui-unit base fallback and register it with the merge-base step test
* ci(unit): keep the setup ceiling at 35 minutes, the install step never took over 33 seconds
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make four flaky unit tests deterministic
The skill archive fixtures stamped the current time into zip bytes used as parametrize ids, so xdist workers collected different ids and errored the whole shard. They now use a fixed timestamp.
The usage-based v2 override test pinned no clock, so a minute boundary between rpm checks reset the counter. It now pins get_utc_datetime the way #49505 does.
The stalled-burst DB lookup test asserted a 2s wall-clock bound. It now asserts every lookup was waiting at once, which fails if lookups are serialized.
The long tool_search placement test asserted a 1s wall-clock bound. It keeps the placement assertion and drops the bound, as #49505 did for the json accumulator
* test(skills): drop the archive fixture timestamp change now that #50118 removed its test
* fix(auth): align cloud credential loading and expiry with the Python SDKs
* feat(auth): resolve consumer-scoped default identities for AWS and Azure
AwsAuthService::default_chain_credentials and AzureAuthService::identity_token give a consumer the SDK's own default identity without LiteLLM's Bedrock or Azure OpenAI settings. Moved here from #49510 so every shared auth crate change lives in one PR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(auth): include identity hints in secret snapshots
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(billing): support Cognito bearer auth for metrics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refetch billing token on 401 and require https
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): import Unpack from typing_extensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): make billing metrics bearer only and drop mtls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format billing metrics module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The bridge moves leading system messages into the Responses instructions
field. With response_format json_object, Azure requires the word json in
input and ignores instructions, so a request that follows the documented
rule 400s after translation. Send the system text as a system input item
when json_object is requested, the same shape already used for
system-only requests
Resolves LIT-10426
* chore(cost-map): set openai gpt-5.6-cyber and Daybreak Red max input to 272K and source the sora-2-pro-high-res deprecation
* Absorb GovCloud Bedrock, Azure GPT-6 Flex and GMI registry rows
Roll the verified registry data and Azure Flex cost coverage into the rolling price sync
Co-authored-by: kerry-berri <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: dguthruf <102602010+danielguthruf@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
