LiteLLM_AccountingEvent and LiteLLM_AccountingDelivery have had no reader since
the SpendLogs projection replaced them, so the migration drops them with their
enums. AccountingDeliveryState stays because LiteLLM_SpendLogDelivery uses it
The schema now spells the delivery_id default the way Postgres stores it, which
ends the drift Prisma reported against the ledger migration. The schema check
compares base types only, so an attribute-only change is no longer reported as
a type change
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(anthropic): stop forcing the response_format json tool when Claude will think
A forced tool call skips Claude's thinking. The json_schema emulation forced
json_tool_call unless thinking.type was enabled, so adaptive thinking and the
models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on
families) lost their thinking on every structured-output request.
AnthropicModelInfo.response_format_tool_can_be_forced is now the one check
the Anthropic, Bedrock Converse (including the native output fallback) and
Databricks paths use. It leaves the tool unforced when thinking is requested
in any form or the model thinks by default, and keeps forcing it when the
request turns thinking off (thinking disabled or reasoning_effort none) on a
model that accepts forced tools.
The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and
Haiku 5.5 on the providers that reach these paths, and a miss-only fallback
rule covers unmapped Claude 5 and later ids.
* fix(bedrock): route native output and thinking token counts by whether Claude will think
Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them.
* fix(anthropic): tell Claude to answer through the unforced response_format tool
Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output
* fix(anthropic): keep open-object schemas on the json tool when Claude will think
Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main
* fix(anthropic): hold streamed prose until the JSON tool decides the answer
In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply.
* test(anthropic): build the stream test events without nested comprehensions
* fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only
* test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model
* fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced
Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse
json_schema stream with the unforced JSON tool never sends Claude's working text as content
* fix(anthropic): drop the thinking False forcing case that completion never reaches
litellm.completion turns thinking False into no thinking param before any route sees it,
so the case was dead on the request path and False keeps meaning the model's default
* test(bedrock): collect the Converse json_schema stream chunks in one assignment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): show model deprecation banner on the models page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh model deprecations when models change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): address review feedback on model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover deprecation role gate and agent builder refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): separate deprecation banner toggle from alerts link
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): dedupe and allow dismissing the model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh table once and key banner entries on successor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: quiet successor logs, count banner models, use local dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): pin banner test date and bucket upcoming entries locally
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover mixed-successor log level through the deprecations endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): move deprecation banner cases into integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep deprecation banner when a refetch fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_map): fill missing Claude capability fields from the family baseline
Mapped Claude entries on Anthropic, Azure Foundry, Bedrock and Vertex now get
supports_parallel_function_calling and supports_system_messages from the
claude-family-baseline rule when the entry leaves them out. Values set on an
entry still win
* test(cost_map): keep the exact fill-missing assertion for Claude thinking rules
Build the expected baseline part from the loaded map so a rule that leaks an extra key into Claude entries still fails, and wrap lines over 120 characters.
* fix(nvidia_riva): send the api key once per call by talking to Riva over grpc directly
Drops nvidia-riva-client from the stt-nvidia-riva extra. Depending on its version it sent
the bearer token zero times on plaintext or twice on TLS, so token-checking Riva deployments
rejected every request (#50059). The provider now declares the few Riva messages it uses,
opens its own grpc channel and puts authorization and function-id on each call exactly once
* test(nvidia_riva): run the provider against a real grpc server built from nvidia's own definitions
* fix(nvidia_riva): validate endpointing settings with pydantic before building the riva message
* test(nvidia_riva): check the key is sent once over tls, and drop a comment that restated the code
* feat(accounting): add opt-in SpendLogs ledger delivery
* fix(accounting): preserve missing organization daily spend parity
* fix(accounting): preserve ledger receipts when replaying migration
* fix(accounting): preserve callback windows and tune ledger batches
* fix(accounting): preserve window charges and bound ledger statements
* test(accounting): verify budget callback registration order
* test(accounting): verify captured charges after repeated startup
* feat(accounting): store durable payloads independently of spend logs
* fix(accounting): preserve first delivery and trace ledger writes
* fix(accounting): keep SDK imports independent of proxy tracing
* fix(accounting): honor log retention during batch takeover
* test(accounting): cover retained batch alerts and invalid settings
* test(accounting): verify downgrade command against pending work
* fix(accounting): drain retries without starving fresh deliveries
* test(accounting): cover delivery projection, metrics and window key reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): read delivery metrics without the shared registry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): charge a leftover $0 batch row when its claim fails
A failed batch claim buffers the retrieve's row and reports it charged. At flush the
row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back
and the charge never reached the spend tables. The flush now takes that row over in the
same transaction and keeps the delivery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): trace the batch takeover at flush as a SpendLogs update
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client
The export client is never created, so main always writes SpendLogs to the proxy's own
database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was
set, which dropped those request logs. It now treats a flush as an export only when the
client exists, and batch claims write local rows as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep every request's charge when its log row is shared, evicted or rejected
- Key non-batch deliveries on a per-request delivery_id so a reused client call id is still
charged once per request, while buffer replays stay deduplicated
- Persist the redacted api_key instead of an opaque bearer token
- Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it
- Strip NUL from payloads and skip non-text tags so projection never parks a paid request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch
A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep charging a request whose tag is too long for the daily tag index
A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): store only guardrail billing fields in the accounting payload
The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): move spend payload stamping out of update_database
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): restore the modules the import-layering guard evicts
The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): keep update_database's payload stamping on its own local
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): type guardrail entries and use a public key redaction helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep charging requests whose end-user id is too long for an index
An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): type the spend log queue snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): leave out any projected row Postgres cannot index or store
Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep the guardrail fields compression savings are counted from
Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): cap projected counters so daily totals cannot overflow
Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): let a buffered batch charge take over its zero-cost receipt
A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read a rescued batch row's spend from its accounting payload
Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read budget windows through the delivery transaction
Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): retry a transient failure while keeping a dropped row's charge
The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): give delivery projection payloads their charged entities
project() now reads entities only from accounting_entities, so the hand-built
delivery payloads carry them like real stored payloads do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): own ledger rows by their payload and key batches on the batch id
Drop the accounting_path stamp and its SpendLogs column. A buffered row that
carries an accounting payload is ledger owned, one without it is legacy, and
spend counters get a plain windows_from_ledger flag. Window seeding tells
applied ledger rows apart by their delivery row.
A ledger batch is plain transaction A keyed by the batch id, so a repeated
completed retrieve conflicts and charges nothing. A $0 retrieve records no
delivery, a batch a legacy retrieve already charged is not charged again, and
the legacy batch path is unchanged from main apart from refusing a batch the
ledger already owns. Batch takeover and its delta bookkeeping are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): drop a projection test duplicated while rebasing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): drop previous_payload now that deliveries are never replaced
Batch takeover was the only writer of previous_payload, so the worker applies
each delivery's payload as stored and the column leaves the migration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): drop the ledger downgrade checklist script
The use_spend_logs_ledger flag is the rollback path, so a separate downgrade
gate is no longer needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): name the window seed reads so their Prisma span resolves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): remove the unused accounting event types and protocols
types.py, protocols.py and measurements.py were left from the old event
scaffolding. Nothing outside the package imports them, so they go with their
tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): park only deliveries whose own data fails
A delivery that keeps failing on a database outage, lock timeout or deadlock
stays pending and retries with capped backoff, so a recovered database applies
it. Only a data error or an invalid payload parks the isolated row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate a spend log row the insert statement rejects
The raw SpendLogs insert validates columns and JSON before Postgres sees the
row, so a row that fails there is now bisected out and dropped alone while its
charge is kept, instead of failing and replaying the whole flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): carry a job's model window on every completed poll in ledger mode
The ledger keeps the poll whose delivery claims the job, which need not be the
poll that won the limiter's in-memory charge. Every completed poll now carries
the job's window, so the delivery that lands always has it, while the live
model budget is still charged once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): leave model-budget windows on the existing window writer
Model-budget windows are written by the model budget limiter rather than
derived from a request's spend, so they stay on main's window queue like live
budget counters do. This drops window capture from the accounting payload,
the deferred limiter registration and the job-window capture, and the ledger
keeps owning key and team budget windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep a delivery pending when the database is unreachable
Prisma reports a can't reach database server outage as a DataError, so the
worker now checks for an unavailable database before treating a data error
as the delivery's own fault.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate only rows the raw insert cannot encode
Catching every ValueError also swallowed unexpected insert errors that main
re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column
or non-JSON value, and only that is isolated as a rejected row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): store zero savings when caller data makes them uncomputable
A huge caller-supplied tokens_saved overflowed the savings math inside
build_accounting_payload, which dropped the request's charge in ledger mode.
Savings that cannot be computed are now recorded as zero and the charge lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows
Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts.
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03
A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503
* fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again
Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(azure): strip output-only reasoning content from Responses input
* test(azure): cover idless encrypted reasoning replay
* fix(azure): preserve existing status handling during reasoning replay
* fix(azure): scope reasoning content filtering by provider
* fix(azure): drop status and content from replayed reasoning items
* fix(azure): forward non-object input items and drop reasoning content on every Azure route
* fix(azure): drop replayed reasoning content only under drop_params
Reasoning status is still dropped on every Azure route. Caller-set reasoning
content now follows the created_by convention from #48783: it is dropped when
drop_params is on and forwarded otherwise, so Azure answers its own 400.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass answered overload status to failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve Anthropic route exception handling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): bind the transformed failure without rebinding the parameter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe the overload failure hook through a real ProxyLogging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(least-busy): reserve the picked deployment so a burst of requests spreads out
The in-flight count only went up in log_pre_api_call, which runs after
selection and the pre-call hooks. Requests selected in that window all saw
the same counts and landed on the same deployment (16 simultaneous
requests over 4 deployments went as lopsided as 12/4/0/0 in the proxy).
When the router selects for a request it is about to send, least-busy now
records a reservation for the chosen deployment in the same synchronous
step that reads the reservations, and log_pre_api_call turns it into the
normal in-flight count. Selections without request kwargs (token counting,
probes) do not reserve. A reservation whose request never reaches the API
call expires after 30 s. The shared and per-worker counts and their
success/failure decrements are unchanged.
* fix(least-busy): prune expired and empty selection reservations
The reservation map kept a key per model group forever, so a wildcard
route with caller-chosen model names grew it without bound. Drop expired
and empty entries on every write.
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
* fix(least-busy): keep pick reservations in the shared router cache
Selectors that share a router cache dedupe to one registered callback, so
only that instance saw log_pre_api_call. Reservations held on the other
instances were never released and skewed picks for the full TTL. Storing
them in the router cache's in-memory layer lets any selector release them,
and the per-key TTL replaces the full-map rebuild on every write.
Tests now hold the burst with a barrier instead of a sleep and use a manual
clock object.
* fix(least-busy): restart the reservation TTL on every write
InMemoryCache keeps the TTL a live key was first given, so a reservation
key that kept being rewritten still expired 30 s after its first pick and
took the newer reservations with it. Drop the key before each write so the
TTL counts from the latest one, and leave it deleted once it is empty.
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
* fix(router): count least-busy decisions calls and release reservations on cache hits
Least-busy now reads the model group from litellm_metadata before metadata, so /v1/decisions calls that carry caller metadata are counted in flight and release their reservation at pre-call. A response cache hit releases its reservation instead of decrementing another request's in-flight count, and Responses WebSocket turn admission selects without reserving. The reservation store is reached through a typed protocol.
* fix(decisions): count proxy /v1/decisions calls in least-busy's in-flight totals
* fix(least-busy): keep reservations out of the shared router cache and count requests that start during a Redis read
Least-busy now reads a call's deployment from the routing metadata the router sets, so /v1/decisions logging params stay as on main and failed decisions calls no longer start cooldowns.
* style(router): wrap the least-busy reservation store line
* fix(least-busy): lock the reservation read-and-update so threaded sync calls keep each other's picks
Router.completion from a thread pool and sync success callbacks can read and rewrite a deployment's reservations at the same time, so one thread's pick or release could be lost.
* fix(least-busy): release a pick whose call ends before reaching the provider
A call that fails or returns before the provider handoff (a 400 for an unsupported param, token counting, a proxy-gate rejection without a start log, batch create) now drops its reservation instead of decrementing a count it never added. Router.select picks and batch cancel no longer reserve, since neither makes a call that logs.
* test(least-busy): give callback fixtures a fixed call-start time
The routing-group tests fed least-busy a finished call without the start time litellm stamps before counting it, so the count stayed up after the fix. Fixtures now use a fixed timestamp instead of the clock.
* fix(least-busy): free tier-group picks, stop lookups reserving, and count only started attempts
Reservations are keyed by deployment id, so a complexity, auto, adaptive or
quality router frees its tier pick when the call starts. Each started attempt
is recorded under its call id, and only a recorded start lowers the count when
the attempt ends, so a fallback hop rejected before the provider frees its pick.
Whether a pick reserves now comes from a context variable set by the routed
call sites, so get_available_deployment and the deprecated
async_get_available_deployment no longer hold a deployment, and custom routing
strategies and Router subclasses keep their method signatures.
* fix(least-busy): free a request's unstarted picks when it ends
Each pick now carries its call id, so a start frees only its own call's pick, and the router frees every pick a request never started once that request ends. A fallback hop rejected before the provider no longer leaves its pick reserved for 30 seconds.
* fix(router): free a least-busy pick when a vector store create ends
* fix(router): free least-busy picks of calls made without a call id
* fix(router): free the least-busy pick of a sync text completion
* test(router): use fixed start times in least-busy fixtures
* fix(router): free the least-busy pick of a sync generic call that fails before its provider
* fix(router): read in-flight counts and start attempts under one lock so a busy deployment never looks free
* fix(router): keep pass-through lookups from reserving least-busy picks
* test(least-busy): wait for the helper thread to start before the timed join
* fix(least-busy): keep realtime sessions out of in-flight counts
---------
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill Batch cached tokens at the Standard cache price when a row has no Batch cache price
* fix(cost): resolve the Standard cache price for the prompt length in the Batch fallback
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GitHub only issues Copilot tokens through copilot_internal/v2/token to its own
Copilot apps, so a custom GITHUB_COPILOT_CLIENT_ID signed users in and then
failed with no Copilot access. With a custom client ID, look up the account at
copilot_internal/user for the seat and Copilot API address and send the user's
GitHub token straight to Copilot. The default client ID keeps the token swap.
* ci(unit): fetch the merge base by sha for the db-and-spend schema check instead of cloning full history
* ci(unit): read the schema check arguments through one typed mapping
* ci(unit): give the schema check its own step with the ui-unit base fallback and register it with the merge-base step test
* ci(unit): keep the setup ceiling at 35 minutes, the install step never took over 33 seconds
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make four flaky unit tests deterministic
The skill archive fixtures stamped the current time into zip bytes used as parametrize ids, so xdist workers collected different ids and errored the whole shard. They now use a fixed timestamp.
The usage-based v2 override test pinned no clock, so a minute boundary between rpm checks reset the counter. It now pins get_utc_datetime the way #49505 does.
The stalled-burst DB lookup test asserted a 2s wall-clock bound. It now asserts every lookup was waiting at once, which fails if lookups are serialized.
The long tool_search placement test asserted a 1s wall-clock bound. It keeps the placement assertion and drops the bound, as #49505 did for the json accumulator
* test(skills): drop the archive fixture timestamp change now that #50118 removed its test
* feat(billing): support Cognito bearer auth for metrics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refetch billing token on 401 and require https
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): import Unpack from typing_extensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): make billing metrics bearer only and drop mtls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format billing metrics module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The bridge moves leading system messages into the Responses instructions
field. With response_format json_object, Azure requires the word json in
input and ignores instructions, so a request that follows the documented
rule 400s after translation. Send the system text as a system input item
when json_object is requested, the same shape already used for
system-only requests
Resolves LIT-10426
* chore(cost-map): set openai gpt-5.6-cyber and Daybreak Red max input to 272K and source the sora-2-pro-high-res deprecation
* Absorb GovCloud Bedrock, Azure GPT-6 Flex and GMI registry rows
Roll the verified registry data and Azure Flex cost coverage into the rolling price sync
Co-authored-by: kerry-berri <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: dguthruf <102602010+danielguthruf@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
* fix(azure): answer 401 when Entra rejects the deployment's credential
A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500.
* fix(azure): keep network failures to Entra a retryable 500
azure-identity wraps a transport failure in ClientAuthenticationError, so the
401 mapping now skips errors whose cause chain holds ServiceRequestError or
ServiceResponseError
* fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials
* fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown
An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP
status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or
throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a
value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch
maps the same rejection, and the A2A endpoint answers it with HTTP 401.
* fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown
* fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections
ClientSecretCredential drops Entra's HTTP status, so throttling and transient
codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable
instead of mapping to 401.
The A2A endpoint answers 401 only for an AuthenticationError caused by an
Entra token rejection, and the router skips cooldown for a caller-sent Entra
field only on Azure-family deployments. The azure/ Decisions route and the
Bing grounding search tool now map an environment-credential rejection to the
same 401.
* fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange
Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents
* fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting
The fallback hop skips cooldown when Entra rejected a credential the caller sent,
Prometheus counts a deployment's own Entra rejection as a deployment failure again,
and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router.
* fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message
A caller-sent Entra field now skips the deployment's cooldown only when it
differs from the deployment, its named credential, the router defaults, the
AZURE_* environment variables and the default scope. An empty value counts as
unset, so a deployment whose own secret is wrong still cools down when the
caller's field changes nothing about the token request.
The /azure passthrough's Entra 401 message goes through the same internal
hostname redaction as other client-facing errors.
* fix(router): compare a caller's Entra field with the value the deployment uses
A caller-sent Entra field skipped the cooldown only when it matched no
configured source, so a value equal to an environment default the deployment
overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own
and let a caller's rejected credential cool down a working deployment.
The check now resolves the value the deployment would use without the caller,
in order from the router defaults, the deployment, its named credential, the
AZURE_* environment variables and the default scope, and skips the cooldown
only when the caller's value differs from it.
* fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged
* fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider
* fix(azure): keep the streaming Entra rejection inside the failed attempt
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): check Anthropic-format request tools in tool_permission pre_call
The pre_call hook only collected tools shaped as {"type": "function", "function": {...}},
so flat tools from /v1/messages ({"name", "input_schema"}) and /v1/responses
({"type": "function", "name"}) were never checked and deny rules silently passed.
Reuse the shared anthropic_tool_names helper so every request tool name is checked,
and handle Anthropic tool_choice {"type": "tool"} in both block and rewrite modes.
Fixes#40583
* fix(guardrails): detect Responses API named tool_choice in tool_permission
Responses API named choices put the name at the top level ({"type": "function", "name": ...}),
so rewrite mode removed the denied tool but left a tool_choice pointing at it.
* test(guardrails): run tool_permission pre_call tests through real default_on gating
Build the guardrail as an always-on pre_call guardrail instead of patching should_run_guardrail.
* test(guardrails): type the new tool_permission pre_call test parameters
* fix(guardrails): keep the Responses API custom tool type distinct
Anthropic's user-defined tools carry type "custom", so pre_call mapped
that to "function" for tool_type rules. The Responses API has its own
custom tool type, and mapping it let a custom tool slip past a
tool_type deny rule when the default action is allow.
Normalize only for the formats where custom means a plain function, and
read tool names directly in the rewrite path, which never needed a type.
* test(guardrails): cover flat tools through Messages and Responses
Add blocked and permitted flat-tool cases through the running proxy, with
endpoint coverage and metadata for the current e2e harness
Require a guardrail policy verdict when checking denied tools so an
unrelated provider error cannot satisfy the regression test
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
* fix(guardrails): check namespace and nameless tools and relax tool_choice in tool permission rewrite
* fix(guardrails): check forced, grouped, loaded, and hosted tools by their real type
* fix(guardrails): drop references to removed tools and count loaded tools in tool permission rewrite
* fix(guardrails): check tools sent outside top-level tools in tool permission
* fix(guardrails): judge namespace members by their bare name as a fallback and keep rewrites valid for Gemini, Bedrock and hosted tools
A namespace member is checked by namespace__name first, and by its bare name when no rule names the joined one. Rewrite mode now prunes Gemini allowedFunctionNames and turns off forced calling when no function is left, drops a removed hosted tool from allowed_tools, reads Bedrock toolSpec and systemTool blocks by name, skips cachePoint markers, and leaves a tools field that is not a list for the provider to reject instead of failing with a 500
* fix(guardrails): let only name rules decide a namespace member's bare-name fallback
* fix(guardrails): judge malformed tool types, empty Gemini name lists and pass-through Anthropic tools like the endpoints do
A tool or tool_choice whose type is not a string now reads as untyped instead of crashing the guardrail with a 500, and a null type key is never read as a Gemini tool part. An empty allowedFunctionNames list no longer turns off the functions a rewrite kept. An Anthropic custom tool reads as a function by its input_schema on every route, so a pass-through route judges it the same as /v1/messages. A Responses WebSocket frame now forwards the extra_body tools a rewrite stripped
* fix(guardrails): judge a named hosted tool by its type and keep a nested chat custom tool custom
* fix(guardrails): keep judging Anthropic client tools and type-named server tools by their name
* fix(guardrails): judge a named tool by its name alone only when the name is its type's own
* fix(guardrails): judge the function a tool names beside a declaration list or a top-level name
* fix(guardrails): judge Gemini hosted keys whatever the tool type and a namespace's function
* fix(guardrails): judge legacy functions entries like tools and send Gemini a namespace's members
* fix(guardrails): judge a namespace member by every name a provider reads and send Gemini its schema converted
* fix(guardrails): judge a legacy functions entry as a function and relax a tool_choice in extra_body
* fix(gemini): keep a namespace whole when inlining its members would repeat a function name
* fix(guardrails): judge a namespace member by every bare name that shares its qualified name
* fix(vertex_ai): declare a repeated namespace member name once instead of sending namespace names to Gemini
* fix(guardrails): judge a tool_choice sent in extra_body like a top-level one
A forced name only extra_body.tool_choice carried was never checked, so block mode let a denied name through to the provider
* fix(guardrails): judge a Gemini namespace member returned by its bare name and force a named namespace's members
* test: keep formatter churn out of the tool_permission and Gemini test files
* fix(guardrails): judge extra_body.input tools, argument-aware alias rules, and exact forced Gemini functions
Tools loaded in extra_body.input are now checked and stripped like top-level input. A returned
namespace member is judged by the alias whose first name rule its arguments satisfy. A forced
Gemini name that matches a top-level function keeps that function. Alias grouping and Gemini
declaration dedupe now run in one pass instead of quadratic scans.
* fix(guardrails): judge and rewrite every tool field sent inside extra_body
extra_body is merged into the upstream body, so functions, function_call,
mcp_servers and toolConfig nested there reached the model unjudged. Treat
extra_body as a second request layer that is collected and rewritten the
same way as the top-level request.
* fix(guardrails): judge Gemini hosted keys by key and count only the tools a provider receives after extra_body
* fix(guardrails): relax a tool_choice that names a dropped namespace member by its bare name
Gemini and Anthropic chat send namespace members under their bare names, so rewrite mode now treats a dropped member's bare name, and an emptied namespace's own name, as removed when no kept tool still carries it
* fix(gemini,guardrails): keep a top-level function over a same-named namespace member and leave malformed tool values to the provider
Gemini and Vertex chat now declare a top-level function over a namespace member that shares its name, so a forced call gets the top-level schema. post_call judges a returned name a top-level tool carries as that tool, and pre_call no longer raises on an unhashable or non-string tool name, input item type, or function_call name.
* fix(guardrails): judge a returned namespace member named like a hosted MCP server as that member
* fix(gemini,guardrails): send a namespace as main did when another tool has a member's name
Gemini chat inlines a namespace's members only when no other tool, namespace
member, or legacy functions entry declares the same name, and otherwise sends the
namespace by its own name as main did. A tool_choice naming such a namespace
forces the namespace itself. The tool permission post_call reads top-level names
the way Gemini declares them and judges a call the Responses bridge tags with a
namespace as that member.
* fix(gemini): send colliding namespace members as namespace__name functions and judge them in post_call
When a namespace member shares a name with another tool, Gemini chat now gets that namespace's members as namespace__name functions instead of one function named after the namespace, so every function it can call is a name the rules judged. Declared names are counted once, and a forced member maps to each holder's qualified name.
post_call judges a returned namespace__name as that member, by its qualified or bare name the way pre_call does, so a bare-name allowlist or denylist applies to it.
* fix(gemini): keep generated member names unique and let a top-level function own its exact name
A namespace that listed a member twice, or a generated namespace__name that
repeated a top-level function, sent Gemini two declarations with one name, and
Gemini answered 400 where main answered 200. A name a namespace lists twice is
now one member, and a top-level function keeps its exact name.
post_call now judges a returned name a top-level function declares as that
function, never as the namespace member the name could also mean.
* fix(gemini): qualify namespace members by the namespace's top-level name
The guardrail judges a member as `name__member` from the namespace's top-level
name. Gemini built the name from `function.name` when the tool also carried one,
so a request could pass pre-call and reach Gemini under a name no rule judged.
* fix(vertex_ai): force only the namespace members Gemini is sent
A forced namespace or member on Gemini chat could still allow a generated name that a top-level function or an earlier namespace owns, so tool_choice forced an unrelated function. The forced names now come from the same collision checks the declarations use
* fix(vertex_ai): reject a forced namespace Gemini gets no member of, and resolve forced names in one pass
* fix(guardrails): judge a namespace member by a rule on its namespace's name
---------
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed
A custom video provider can report a video's cost on the status poll that
sees it completed. Every poll of a video shared the video id as its spend
log request_id, so the paid poll's row was dropped as a duplicate and every
later poll of the finished video charged the cost again.
Video status rows now carry their own `_video_cost` request_id, and a poll
that reports a cost goes through the same settle-once claim batches use, so
the first poll that sees the video completed charges it and writes the row,
and every later poll charges nothing.
* fix(proxy): charge a video's provider-reported cost once on model, router and session budgets too
A video status poll gets the _video_cost row suffix only when it reports a cost, so a $0 poll keeps the bare video id and adds no row of its own, and the spend log writer claims a row by that suffix.
The batch charge-once marker now covers videos, keyed on the video id under {spend_key}:video:{id}, with the batch marker key unchanged. Model and router budgets use it, and per-session budgets now charge a repolled batch or video once and ignore a cost that is not a finite number
* fix(proxy): charge a finished video's own cost once, keyed by the id the client polled
Guardrail cost on a running poll no longer takes the charge-once claim, only a completed poll charges, the session marker TTL refreshes like the others, and two deployments whose provider ids match each keep their own charge.
* fix(proxy): charge repeat poll guardrails, key video charges by canonical id and survive redaction
* fix(proxy): keep video status polls in-process for the collector and give guardrail-only job polls an unclaimable row id
* fix(proxy): keep repeat batch polls free and group guardrail-only batch rows under their batch's trace
* fix(proxy): settle a finished job whose own cost is below its poll's guardrail cost under the job's charge row
* test(proxy): count charged_poll_cost as covered through the finished job guardrail poll tests
* fix(proxy): settle a finished batch that cost nothing itself on its first priced retrieve
* fix(proxy): charge no budget for a batch retrieve before its cost is final, and read the retrieved batch from before redaction
* fix(proxy): charge sessions a poll's guardrail-inclusive cost, count a job cost row's guardrail run, and keep batch_cost_is_final importable from litellm.batches
* fix(proxy): claim a job and charge the session budget in one Redis call
The session limiter claimed the job marker and then raised spend in a second Redis call. A failure between the two left the job claimed while its charge sat only in local memory, so later polls skipped it. One script now does both, and a refused script claims and charges locally
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: update EmpirioLabs models in model_prices_and_context_window.json
* chore(endpoints): regenerate provider endpoint support for the EmpirioLabs cost map rows
* fix(empiriolabs): drop the retired max_tokens field, route embeddings and remove the unrouted rerank row
The cost map test rejects any row carrying max_tokens, so every EmpirioLabs row keeps max_input_tokens and max_output_tokens only. The provider JSON now lists /v1/embeddings so embedding(model="empiriolabs/text-embedding-v4") reaches the OpenAI-compatible embeddings route instead of failing provider lookup. The qwen3-rerank row goes away because no rerank route exists for the provider. Endpoint support regenerated from the cost map
* fix(empiriolabs): mark audio input and image embedding input on the rows the provider lists as multimodal
The three vision embedding rows get supports_embedding_image_input and the eleven chat rows whose input_modalities on GET https://api.empiriolabs.ai/v1/models include audio get supports_audio_input, so capability checks stop treating them as text only
* fix(empiriolabs): carry the long-context price tiers on the 24 rows the provider bills in bands
EmpirioLabs bills 24 of these models at a higher rate once the prompt passes 32K, 128K, 256K, 272K or 512K tokens, and the rows carried only the first band, so a 300K-token qwen3-7-plus request was priced at a third of the real cost. Each of those rows now carries input_cost_per_token_above_<N>k_tokens, output_cost_per_token_above_<N>k_tokens and, where the row already has cache_read_input_token_cost, cache_read_input_token_cost_above_<N>k_tokens for every band, taken from pricing[].min_context on GET https://api.empiriolabs.ai/v1/models for the base rows and from the Variants rate card on docs.empiriolabs.ai/models/<id> for the six variant rows
* feat(empiriolabs): bill the per-call web search fee reported in usage.tool_usage
EmpirioLabs charges a flat fee per invoked web_search, web_search_image or image_search call and reports the counts in usage.tool_usage on the chat response. The JSON provider config now carries that count as usage.server_tool_use.web_search_requests when providers.json sets special_handling.web_search_requests_from_tool_usage, the per-call web search cost path bills the usage-reported count instead of flooring to one call, and the 39 EmpirioLabs rows the provider lists a per-call search price for carry it as search_context_cost_per_query, from unit_pricing on GET /v1/models for base rows and the Variants rate card on docs.empiriolabs.ai for variant rows
* fix(openai_like): mark the dict signature of transform_parsed_response_dict as matching BaseConfig for LIT001
* fix(openai_like): mark both dict annotations of transform_parsed_response_dict for LIT001
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): stamp the failed deployment on every endpoint's attempt
Weighted failover (#27980) re-picks within a model group by excluding the
deployment named in the error's failed_deployment_id. #27980 stamped that id
in only three attempt functions: _completion, _acompletion and
_ageneric_api_call_with_fallbacks_helper. Embedding, image generation,
transcription, speech, text completion, rerank, adapter completion, the sync
generic call and vector store create never stamped it, so with
enable_weighted_failover on, a failure on one of their deployments skipped the
in-group re-pick and went straight to the cross-group fallbacks, or raised.
Each of those attempts now stamps the deployment it picked, read from the
model_info that _update_kwargs_with_deployment put on the request (so a
client-side-credential call names its own dynamic deployment, as on
completions). A failed pick stamps nothing. File and batch creation and batch
cancellation are left alone: they are bound to a resource on the deployment
they picked, so re-picking another one cannot help.
* test(router): narrow the failed-pick raise to the no-healthy-deployments error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013FDwtoMjnHa4xxazGJJNSm
* fix(router): exclude the configured deployment on failover for client-side credentials
Weighted failover excluded the per-request id that client-supplied credentials
generate, so the failed configured deployment stayed eligible for the re-pick.
Stamp the configured id alongside it and exclude that. Also drop the vector
store create stamp, which no fallback path reads, and redundant test comments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013FDwtoMjnHa4xxazGJJNSm
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(decisions): add EmpirioLabs AI as a decisions provider
empiriolabs/aplomb-1 now works with litellm.decisions and litellm.systemone, the proxy /v1/decisions and /v1/systemone routes, the decision model guardrail and the Admin UI. Requests go to POST /v1/decisions on the EmpirioLabs AI API with the System One body, keyed by EMPIRIOLABS_API_KEY with an optional EMPIRIOLABS_API_BASE
Chat models keep routing through the existing OpenAI-compatible JSON provider, and a test pins that
* ci: run the EmpirioLabs AI decisions tests in the llms unit shard
* fix: align EmpirioLabs registration with CI gates
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(accounting): replace event scaffolding with SpendLogs projection
* test: preserve spend timestamps without removed capture callback
* fix: guard accounting event removal across schema update paths
* refactor(accounting): retain schema and prepare durable payloads
* fix(accounting): preserve dates in serialized payloads
* fix(accounting): retain schema push compatibility
* fix(accounting): retain span registration for the legacy schema
* test(accounting): remove obsolete capture acknowledgement test
* fix(accounting): date every daily projection row the way the model rollup does
A space-separated startTime kept the whole timestamp as the daily key, while the model usage rollup took the calendar date. Every daily row now uses the same date helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore(ui): regenerate dashboard API types after rebasing on main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): project only from the stored accounting payload
project() now reads entities only from the payload's accounting_entities, so
the raw SpendLogs fallback branches are gone. build_accounting_payload() builds
the metadata once instead of dumping it to JSON and parsing it back. Guardrail
usage dates use a tolerant UTC parse, so a malformed startTime drops only the
guardrail rows instead of failing the whole projection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: prevent reasoning field from being copied into provider_specific_fields
* fix: ensure reasoning field in provider_specific_fields is redacted
* fix: redact reasoning_details in provider_specific_fields
* fix: format redact_messages.py
* ci: retrigger
* ci: retrigger
* ci: retrigger
* fix(logging): keep provider_specific_fields.reasoning in chat responses
Redaction already blanks provider_specific_fields.reasoning and reasoning_details in logs, so chat responses can keep the field as they did before.
* fix(logging): redact reasoning_text, citations, and web search results in provider_specific_fields
* fix(logging): redact vector store, RAG, Groq tool, Databricks agent and tool search results in provider_specific_fields
* fix(logging): redact Bedrock streamed reasoningContent and file annotations in provider_specific_fields
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): retire the quiet_stack lock
quiet_stack held an exclusive host-wide lock for six tests, draining every
other xdist worker while they ran. On the full e2e lane that left 35-38% of
worker time idle. Each of those tests now lives where it can run without
pausing the suite
- the two memory tests move to perf/ behind the perf marker, so the perf lane
runs them on its own fresh stack. Their refusing deployments use a literal
key because the perf stack carries no provider keys
- the stored-request size check from the memory test becomes an integration
test on an owned proxy that stores prompts
- the discount and margin spend-row tests are dropped from e2e. Their
breakdown assertions join the existing integration pricing test, which
already owns the discount and margin repricing flow
- the Bedrock guardrail cost test stays in e2e without the pricing reset
fixture, since nothing else on the stack writes pricing config now
- the MCP semantic tool filter test turns on a proxy-wide setting, so it gets
a new proxy_wide_config opt-in marker and runs after the main run
* test(e2e): register the claude_code deployments as one batch
The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
* feat(openai): bill gov.api.openai.com FedRAMP endpoint with 1.1x uplift
* chore(openai): trim FedRAMP uplift diff
* refactor(openai): bill the FedRAMP endpoint through its own fedramp_uplift_multiplier key
* fix(ui): regenerate fedramp_uplift_multiplier doc comment in schema.d.ts
* style: ruff format utils.py and reorder fedramp_uplift_multiplier in schema.d.ts
* fix(openai): infer data_residency from OPENAI_BASE_URL and litellm.api_base when api_base is unset
* test(cost): cover get_fedramp_uplift_multiplier fallback paths
* fix(cost): apply fedramp_uplift_multiplier in batch_cost_calculator
* fix(openai): derive data_residency from a supplied client's base_url
* test: type the supplied-client data_residency test params
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in proxy sub-call budget checks, model search and logger init
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in otel baggage metadata, model budget limiter and batch line items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "refactor: clean up fresh tech debt in otel baggage metadata, model budget limiter and batch line items"
This reverts commit 68737c457c.
* refactor: clean up fresh tech debt in guardrail init, cost map fetch and stream continuation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in kimi k3 token counting and two slop comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in mcp tool call auth fields, messages drop_params, exception chain and spend log noqa
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve dict-compatible drop params access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in user list sorting, opencode affinity header and aws polly comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in responses bridge check, chatgpt authenticator and image header adapters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in auth 401 detection, cooldown fallback lookup and bedrock guardrail billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "refactor: clean up fresh tech debt in auth 401 detection, cooldown fallback lookup and bedrock guardrail billing"
This reverts commit d497e02bbf.
* refactor: clean up fresh tech debt in responses stream usage cost, bedrock count tokens comments and router return types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in messages drop_params reads, auth zero cost comment and two return types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: keep getattr drop_params reads where callers pass a plain dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: clean up fresh tech debt in router context signatures, search tool comment and gemini realtime init
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: drop fresh private-alias, getattr and duplicated response id helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: bind response id helper to logging instance
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: simplify media guard and spend log reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: satisfy media guard type discipline and formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: keep media guard cast rather than trade it for new suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): rebuild the public AI Hub as a searchable list with a details pane
The public AI Hub at /ui/model_hub_table (and the older /ui/model_hub) now
renders PublicAiHub, built from shadcn parts only: a header with the
gateway's base URL and the admin's links, segmented tabs for models, agents,
MCP servers and skills, and a list beside a details pane on desktop that
becomes a bottom sheet on phones. The open item and tab live in the URL.
It keeps every feature the old page had: server-side search, provider, mode
and feature filters, sorting, paging, health, rate limits, prices, custom
metadata, supported params, usage snippets, the require-auth redirect and
the not-enabled state. cURL shows only for chat models, since the old one
called chat completions for every mode.
The navbar hides the community buttons below the sm breakpoint on every page,
and the public navbar uses a smaller logo on phones, so Docs and Blog no
longer collide with them.
* fix(ui): harden the public AI Hub after adversarial review
- Show "This AI Hub isn't available" only when every hub call fails; a
failed models call now shows an inline error with Try again and leaves
agents, MCP servers and skills usable (a gateway with no models answers
the models list with 400)
- Row links keep the page's other query params, so open-in-new-tab and
copy link no longer 404
- A shared ?open= link resolves models beyond the first page, keeps the
opened item when a search filters it out, and says when it is no
longer listed instead of showing a different item
- Searching, sorting and paging no longer blank the details pane or drop
the Operational badge
- Key MCP servers by their listed name, which is never null
- Keep sub-cent prices and show Free for zero cost, fold evaluation into
Decisions, sort agents, MCP servers and skills by name, restore
word-wise search, and show a skill's source, author and date added
- Drop the invented description for models without a mode, the empty
agent modes row, the stray auth dash on MCP rows and NaN dates
- Do not retry the UI config call, and close the 1023-1024px gap between
the sheet and the details pane
- Update the public hub Playwright spec for the new tabs and list, and
prune the lint suppressions the old route pages needed
* fix(ui): surface hub load errors and keep the open item steady
Agents, MCP servers and skills now load through apiClient so an HTTP error
shows the inline error instead of an empty "published yet" state, and an
all-failing gateway reaches the unavailable page. Try again shows loading
instead of an empty hub, the open model stays in the details pane when a
search or page change hides its row, a zero-result search keeps the pane,
phones get the not-listed sheet for a stale link, an empty open param means
nothing opened, free models read Free in the list, and an invalid skill date
is hidden
* fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call
* fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests
* fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500
* fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message
* fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails
A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever.
A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry.
On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed
* fix(agentic loop): bill retried attempts and clean up what failed or capped turns return
The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend
A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers
Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry
The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints
* fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns
A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call
Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try
* fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served
An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the
router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of
rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run.
The failure row bills every round the loop served before the failure, at any depth, and a later attempt
that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the
response cost headers adding up. The error response's x-litellm-response-cost reports the same amount
its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before.
A refused tool call now matches by id among the refused ids, or by name among refused calls that carry
no id, so a name-only refusal next to an id refusal is no longer kept.
* fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one
* fix(code_interpreter): delete only the session sandbox that failed, never its replacement
* test(code_interpreter): type the session sandbox test helpers
* fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors
* fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success
An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's
plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no
longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too.
A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row.
A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed
agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main.
A code interpreter sandbox that fails while running the code now says the code may have run but its
result is lost, instead of saying the code did not run.
* test(responses): bill a retried MCP request each attempt on its own row again
Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row
* fix(spend): name a failed follow-up attempt the success record already bills
With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off
* fix(spend): count a billed failed follow-up's tokens on the success row
* fix(spend): bill a failed follow-up on a fallback served from cache
* fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown
* fix(spend): keep a failed follow-up's billing state off the request-merged call details
* fix(spend): record the cost breakdown of a successful agentic follow-up
The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): send only shown SSO fields when editing, and keep env-row SSO values behind a blank stored value
* fix(ui): clear every competing SSO provider when the admin switches provider
Only the detected provider's fields were nulled on a switch, so a second stored
provider (e.g. Microsoft behind Google) kept outranking the new one at sign-in
* chore(cost-map): remove the 52 rows whose provider retired them before 2026
* test(utils): pin the proxy function-calling parity case to azure/gpt-4.1
* test(batches): price the Azure batch-logging regression case against gpt-4.1
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
