LiteLLM_AccountingEvent and LiteLLM_AccountingDelivery have had no reader since
the SpendLogs projection replaced them, so the migration drops them with their
enums. AccountingDeliveryState stays because LiteLLM_SpendLogDelivery uses it
The schema now spells the delivery_id default the way Postgres stores it, which
ends the drift Prisma reported against the ledger migration. The schema check
compares base types only, so an attribute-only change is no longer reported as
a type change
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(anthropic): stop forcing the response_format json tool when Claude will think
A forced tool call skips Claude's thinking. The json_schema emulation forced
json_tool_call unless thinking.type was enabled, so adaptive thinking and the
models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on
families) lost their thinking on every structured-output request.
AnthropicModelInfo.response_format_tool_can_be_forced is now the one check
the Anthropic, Bedrock Converse (including the native output fallback) and
Databricks paths use. It leaves the tool unforced when thinking is requested
in any form or the model thinks by default, and keeps forcing it when the
request turns thinking off (thinking disabled or reasoning_effort none) on a
model that accepts forced tools.
The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and
Haiku 5.5 on the providers that reach these paths, and a miss-only fallback
rule covers unmapped Claude 5 and later ids.
* fix(bedrock): route native output and thinking token counts by whether Claude will think
Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them.
* fix(anthropic): tell Claude to answer through the unforced response_format tool
Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output
* fix(anthropic): keep open-object schemas on the json tool when Claude will think
Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main
* fix(anthropic): hold streamed prose until the JSON tool decides the answer
In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply.
* test(anthropic): build the stream test events without nested comprehensions
* fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only
* test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model
* fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced
Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse
json_schema stream with the unforced JSON tool never sends Claude's working text as content
* fix(anthropic): drop the thinking False forcing case that completion never reaches
litellm.completion turns thinking False into no thinking param before any route sees it,
so the case was dead on the request path and False keeps meaning the model's default
* test(bedrock): collect the Converse json_schema stream chunks in one assignment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): show model deprecation banner on the models page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh model deprecations when models change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): address review feedback on model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover deprecation role gate and agent builder refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): separate deprecation banner toggle from alerts link
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): dedupe and allow dismissing the model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh table once and key banner entries on successor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: quiet successor logs, count banner models, use local dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): pin banner test date and bucket upcoming entries locally
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover mixed-successor log level through the deprecations endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): move deprecation banner cases into integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep deprecation banner when a refetch fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_map): fill missing Claude capability fields from the family baseline
Mapped Claude entries on Anthropic, Azure Foundry, Bedrock and Vertex now get
supports_parallel_function_calling and supports_system_messages from the
claude-family-baseline rule when the entry leaves them out. Values set on an
entry still win
* test(cost_map): keep the exact fill-missing assertion for Claude thinking rules
Build the expected baseline part from the loaded map so a rule that leaks an extra key into Claude entries still fails, and wrap lines over 120 characters.
* fix(nvidia_riva): send the api key once per call by talking to Riva over grpc directly
Drops nvidia-riva-client from the stt-nvidia-riva extra. Depending on its version it sent
the bearer token zero times on plaintext or twice on TLS, so token-checking Riva deployments
rejected every request (#50059). The provider now declares the few Riva messages it uses,
opens its own grpc channel and puts authorization and function-id on each call exactly once
* test(nvidia_riva): run the provider against a real grpc server built from nvidia's own definitions
* fix(nvidia_riva): validate endpointing settings with pydantic before building the riva message
* test(nvidia_riva): check the key is sent once over tls, and drop a comment that restated the code
The key edit form read react-hook-form's dirtyFields only inside the submit
handler. formState is a proxy that only tracks what the component reads while
rendering, so on the first save of an edit session that read returned the
initial empty object. A team admin editing a member's key then sent only the
key id to /key/update, got a 200 and a "Key updated successfully" toast, and
nothing changed: a max_budget edit was silently dropped, and a rename went
through as an empty update instead of the "team admins can only change budget
fields" error.
Read dirtyFields during render so the form subscribes to it and the submit
handler sees the fields the user actually changed.
* feat(accounting): add opt-in SpendLogs ledger delivery
* fix(accounting): preserve missing organization daily spend parity
* fix(accounting): preserve ledger receipts when replaying migration
* fix(accounting): preserve callback windows and tune ledger batches
* fix(accounting): preserve window charges and bound ledger statements
* test(accounting): verify budget callback registration order
* test(accounting): verify captured charges after repeated startup
* feat(accounting): store durable payloads independently of spend logs
* fix(accounting): preserve first delivery and trace ledger writes
* fix(accounting): keep SDK imports independent of proxy tracing
* fix(accounting): honor log retention during batch takeover
* test(accounting): cover retained batch alerts and invalid settings
* test(accounting): verify downgrade command against pending work
* fix(accounting): drain retries without starving fresh deliveries
* test(accounting): cover delivery projection, metrics and window key reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): read delivery metrics without the shared registry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): charge a leftover $0 batch row when its claim fails
A failed batch claim buffers the retrieve's row and reports it charged. At flush the
row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back
and the charge never reached the spend tables. The flush now takes that row over in the
same transaction and keeps the delivery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): trace the batch takeover at flush as a SpendLogs update
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client
The export client is never created, so main always writes SpendLogs to the proxy's own
database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was
set, which dropped those request logs. It now treats a flush as an export only when the
client exists, and batch claims write local rows as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep every request's charge when its log row is shared, evicted or rejected
- Key non-batch deliveries on a per-request delivery_id so a reused client call id is still
charged once per request, while buffer replays stay deduplicated
- Persist the redacted api_key instead of an opaque bearer token
- Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it
- Strip NUL from payloads and skip non-text tags so projection never parks a paid request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch
A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep charging a request whose tag is too long for the daily tag index
A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): store only guardrail billing fields in the accounting payload
The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): move spend payload stamping out of update_database
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): restore the modules the import-layering guard evicts
The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): keep update_database's payload stamping on its own local
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): type guardrail entries and use a public key redaction helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep charging requests whose end-user id is too long for an index
An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): type the spend log queue snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): leave out any projected row Postgres cannot index or store
Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep the guardrail fields compression savings are counted from
Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): cap projected counters so daily totals cannot overflow
Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): let a buffered batch charge take over its zero-cost receipt
A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read a rescued batch row's spend from its accounting payload
Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read budget windows through the delivery transaction
Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): retry a transient failure while keeping a dropped row's charge
The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): give delivery projection payloads their charged entities
project() now reads entities only from accounting_entities, so the hand-built
delivery payloads carry them like real stored payloads do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): own ledger rows by their payload and key batches on the batch id
Drop the accounting_path stamp and its SpendLogs column. A buffered row that
carries an accounting payload is ledger owned, one without it is legacy, and
spend counters get a plain windows_from_ledger flag. Window seeding tells
applied ledger rows apart by their delivery row.
A ledger batch is plain transaction A keyed by the batch id, so a repeated
completed retrieve conflicts and charges nothing. A $0 retrieve records no
delivery, a batch a legacy retrieve already charged is not charged again, and
the legacy batch path is unchanged from main apart from refusing a batch the
ledger already owns. Batch takeover and its delta bookkeeping are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): drop a projection test duplicated while rebasing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): drop previous_payload now that deliveries are never replaced
Batch takeover was the only writer of previous_payload, so the worker applies
each delivery's payload as stored and the column leaves the migration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): drop the ledger downgrade checklist script
The use_spend_logs_ledger flag is the rollback path, so a separate downgrade
gate is no longer needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): name the window seed reads so their Prisma span resolves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): remove the unused accounting event types and protocols
types.py, protocols.py and measurements.py were left from the old event
scaffolding. Nothing outside the package imports them, so they go with their
tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): park only deliveries whose own data fails
A delivery that keeps failing on a database outage, lock timeout or deadlock
stays pending and retries with capped backoff, so a recovered database applies
it. Only a data error or an invalid payload parks the isolated row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate a spend log row the insert statement rejects
The raw SpendLogs insert validates columns and JSON before Postgres sees the
row, so a row that fails there is now bisected out and dropped alone while its
charge is kept, instead of failing and replaying the whole flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): carry a job's model window on every completed poll in ledger mode
The ledger keeps the poll whose delivery claims the job, which need not be the
poll that won the limiter's in-memory charge. Every completed poll now carries
the job's window, so the delivery that lands always has it, while the live
model budget is still charged once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): leave model-budget windows on the existing window writer
Model-budget windows are written by the model budget limiter rather than
derived from a request's spend, so they stay on main's window queue like live
budget counters do. This drops window capture from the accounting payload,
the deferred limiter registration and the job-window capture, and the ledger
keeps owning key and team budget windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep a delivery pending when the database is unreachable
Prisma reports a can't reach database server outage as a DataError, so the
worker now checks for an unavailable database before treating a data error
as the delivery's own fault.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate only rows the raw insert cannot encode
Catching every ValueError also swallowed unexpected insert errors that main
re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column
or non-JSON value, and only that is isolated as a rejected row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): store zero savings when caller data makes them uncomputable
A huge caller-supplied tokens_saved overflowed the savings math inside
build_accounting_payload, which dropped the request's charge in ledger mode.
Savings that cannot be computed are now recorded as zero and the charge lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows
Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts.
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03
A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503
* fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again
Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(azure): strip output-only reasoning content from Responses input
* test(azure): cover idless encrypted reasoning replay
* fix(azure): preserve existing status handling during reasoning replay
* fix(azure): scope reasoning content filtering by provider
* fix(azure): drop status and content from replayed reasoning items
* fix(azure): forward non-object input items and drop reasoning content on every Azure route
* fix(azure): drop replayed reasoning content only under drop_params
Reasoning status is still dropped on every Azure route. Caller-set reasoning
content now follows the created_by convention from #48783: it is dropped when
drop_params is on and forwarded otherwise, so Azure answers its own 400.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Every string the Models + Endpoints, Model Insights and Transform Request
pages write themselves now goes through t() or Trans, including the shared
auto-router settings those pages render on other routes. 1,651 English
messages are added under 28 new top-level keys in en.json.
This is a pure text move. The English on screen does not change, option
lists and constants keep their shape, and text that could not move without
restructuring stays English. ModelInfoEditForm.tsx, ComplexityRouterConfig.tsx
and build_complexity_router_config.ts grew past the 800 line limit and get
max-lines entries in eslint-suppressions.json instead of being split.
Move batch B1 of the admin dashboard onto react-i18next: the dashboard
shell (navbar, command palette, deployment health, header, layouts), the
entry pages (login, onboarding, change password, home, API reference,
UI theme, workflows), the public AI hub and the AI hub table, the shared
components and helpers in src/components/shared, common_components,
networking toasts, the toast titles and the shadcn primitive labels.
Pure text move into t() and Trans with 1,056 new en.json keys. The
English on screen is unchanged.
* fix(bedrock): add GovCloud Kimi K3 runtime prices
* fix(bedrock): omit retired max_tokens from Kimi rows
* fix(bedrock): add GovCloud Haiku and Grok prices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): match GovCloud Grok 4.7 limits to the card and date GovCloud Claude 3 Haiku
* fix(bedrock): add priced Grok Mantle base for regional routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass answered overload status to failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve Anthropic route exception handling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): bind the transformed failure without rebinding the parameter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe the overload failure hook through a real ProxyLogging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(least-busy): reserve the picked deployment so a burst of requests spreads out
The in-flight count only went up in log_pre_api_call, which runs after
selection and the pre-call hooks. Requests selected in that window all saw
the same counts and landed on the same deployment (16 simultaneous
requests over 4 deployments went as lopsided as 12/4/0/0 in the proxy).
When the router selects for a request it is about to send, least-busy now
records a reservation for the chosen deployment in the same synchronous
step that reads the reservations, and log_pre_api_call turns it into the
normal in-flight count. Selections without request kwargs (token counting,
probes) do not reserve. A reservation whose request never reaches the API
call expires after 30 s. The shared and per-worker counts and their
success/failure decrements are unchanged.
* fix(least-busy): prune expired and empty selection reservations
The reservation map kept a key per model group forever, so a wildcard
route with caller-chosen model names grew it without bound. Drop expired
and empty entries on every write.
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
* fix(least-busy): keep pick reservations in the shared router cache
Selectors that share a router cache dedupe to one registered callback, so
only that instance saw log_pre_api_call. Reservations held on the other
instances were never released and skewed picks for the full TTL. Storing
them in the router cache's in-memory layer lets any selector release them,
and the per-key TTL replaces the full-map rebuild on every write.
Tests now hold the burst with a barrier instead of a sleep and use a manual
clock object.
* fix(least-busy): restart the reservation TTL on every write
InMemoryCache keeps the TTL a live key was first given, so a reservation
key that kept being rewritten still expired 30 s after its first pick and
took the newer reservations with it. Drop the key before each write so the
TTL counts from the latest one, and leave it deleted once it is empty.
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
* fix(router): count least-busy decisions calls and release reservations on cache hits
Least-busy now reads the model group from litellm_metadata before metadata, so /v1/decisions calls that carry caller metadata are counted in flight and release their reservation at pre-call. A response cache hit releases its reservation instead of decrementing another request's in-flight count, and Responses WebSocket turn admission selects without reserving. The reservation store is reached through a typed protocol.
* fix(decisions): count proxy /v1/decisions calls in least-busy's in-flight totals
* fix(least-busy): keep reservations out of the shared router cache and count requests that start during a Redis read
Least-busy now reads a call's deployment from the routing metadata the router sets, so /v1/decisions logging params stay as on main and failed decisions calls no longer start cooldowns.
* style(router): wrap the least-busy reservation store line
* fix(least-busy): lock the reservation read-and-update so threaded sync calls keep each other's picks
Router.completion from a thread pool and sync success callbacks can read and rewrite a deployment's reservations at the same time, so one thread's pick or release could be lost.
* fix(least-busy): release a pick whose call ends before reaching the provider
A call that fails or returns before the provider handoff (a 400 for an unsupported param, token counting, a proxy-gate rejection without a start log, batch create) now drops its reservation instead of decrementing a count it never added. Router.select picks and batch cancel no longer reserve, since neither makes a call that logs.
* test(least-busy): give callback fixtures a fixed call-start time
The routing-group tests fed least-busy a finished call without the start time litellm stamps before counting it, so the count stayed up after the fix. Fixtures now use a fixed timestamp instead of the clock.
* fix(least-busy): free tier-group picks, stop lookups reserving, and count only started attempts
Reservations are keyed by deployment id, so a complexity, auto, adaptive or
quality router frees its tier pick when the call starts. Each started attempt
is recorded under its call id, and only a recorded start lowers the count when
the attempt ends, so a fallback hop rejected before the provider frees its pick.
Whether a pick reserves now comes from a context variable set by the routed
call sites, so get_available_deployment and the deprecated
async_get_available_deployment no longer hold a deployment, and custom routing
strategies and Router subclasses keep their method signatures.
* fix(least-busy): free a request's unstarted picks when it ends
Each pick now carries its call id, so a start frees only its own call's pick, and the router frees every pick a request never started once that request ends. A fallback hop rejected before the provider no longer leaves its pick reserved for 30 seconds.
* fix(router): free a least-busy pick when a vector store create ends
* fix(router): free least-busy picks of calls made without a call id
* fix(router): free the least-busy pick of a sync text completion
* test(router): use fixed start times in least-busy fixtures
* fix(router): free the least-busy pick of a sync generic call that fails before its provider
* fix(router): read in-flight counts and start attempts under one lock so a busy deployment never looks free
* fix(router): keep pass-through lookups from reserving least-busy picks
* test(least-busy): wait for the helper thread to start before the timed join
* fix(least-busy): keep realtime sessions out of in-flight counts
---------
Signed-off-by: Oliver Yu <tc3oliver@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill Batch cached tokens at the Standard cache price when a row has no Batch cache price
* fix(cost): resolve the Standard cache price for the prompt length in the Batch fallback
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GitHub only issues Copilot tokens through copilot_internal/v2/token to its own
Copilot apps, so a custom GITHUB_COPILOT_CLIENT_ID signed users in and then
failed with no Copilot access. With a custom client ID, look up the account at
copilot_internal/user for the seat and Copilot API address and send the user's
GitHub token straight to Copilot. The default client ID keeps the token swap.
* ci(unit): fetch the merge base by sha for the db-and-spend schema check instead of cloning full history
* ci(unit): read the schema check arguments through one typed mapping
* ci(unit): give the schema check its own step with the ui-unit base fallback and register it with the merge-base step test
* ci(unit): keep the setup ceiling at 35 minutes, the install step never took over 33 seconds
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make four flaky unit tests deterministic
The skill archive fixtures stamped the current time into zip bytes used as parametrize ids, so xdist workers collected different ids and errored the whole shard. They now use a fixed timestamp.
The usage-based v2 override test pinned no clock, so a minute boundary between rpm checks reset the counter. It now pins get_utc_datetime the way #49505 does.
The stalled-burst DB lookup test asserted a 2s wall-clock bound. It now asserts every lookup was waiting at once, which fails if lookups are serialized.
The long tool_search placement test asserted a 1s wall-clock bound. It keeps the placement assertion and drops the bound, as #49505 did for the json accumulator
* test(skills): drop the archive fixture timestamp change now that #50118 removed its test
* fix(auth): align cloud credential loading and expiry with the Python SDKs
* feat(auth): resolve consumer-scoped default identities for AWS and Azure
AwsAuthService::default_chain_credentials and AzureAuthService::identity_token give a consumer the SDK's own default identity without LiteLLM's Bedrock or Azure OpenAI settings. Moved here from #49510 so every shared auth crate change lives in one PR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(auth): include identity hints in secret snapshots
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(billing): support Cognito bearer auth for metrics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refetch billing token on 401 and require https
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): import Unpack from typing_extensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): make billing metrics bearer only and drop mtls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format billing metrics module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The bridge moves leading system messages into the Responses instructions
field. With response_format json_object, Azure requires the word json in
input and ignores instructions, so a request that follows the documented
rule 400s after translation. Send the system text as a system input item
when json_object is requested, the same shape already used for
system-only requests
Resolves LIT-10426
* chore(cost-map): set openai gpt-5.6-cyber and Daybreak Red max input to 272K and source the sora-2-pro-high-res deprecation
* Absorb GovCloud Bedrock, Azure GPT-6 Flex and GMI registry rows
Roll the verified registry data and Azure Flex cost coverage into the rolling price sync
Co-authored-by: kerry-berri <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: dguthruf <102602010+danielguthruf@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
* fix(azure): answer 401 when Entra rejects the deployment's credential
A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500.
* fix(azure): keep network failures to Entra a retryable 500
azure-identity wraps a transport failure in ClientAuthenticationError, so the
401 mapping now skips errors whose cause chain holds ServiceRequestError or
ServiceResponseError
* fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials
* fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown
An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP
status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or
throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a
value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch
maps the same rejection, and the A2A endpoint answers it with HTTP 401.
* fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown
* fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections
ClientSecretCredential drops Entra's HTTP status, so throttling and transient
codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable
instead of mapping to 401.
The A2A endpoint answers 401 only for an AuthenticationError caused by an
Entra token rejection, and the router skips cooldown for a caller-sent Entra
field only on Azure-family deployments. The azure/ Decisions route and the
Bing grounding search tool now map an environment-credential rejection to the
same 401.
* fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange
Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents
* fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting
The fallback hop skips cooldown when Entra rejected a credential the caller sent,
Prometheus counts a deployment's own Entra rejection as a deployment failure again,
and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router.
* fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message
A caller-sent Entra field now skips the deployment's cooldown only when it
differs from the deployment, its named credential, the router defaults, the
AZURE_* environment variables and the default scope. An empty value counts as
unset, so a deployment whose own secret is wrong still cools down when the
caller's field changes nothing about the token request.
The /azure passthrough's Entra 401 message goes through the same internal
hostname redaction as other client-facing errors.
* fix(router): compare a caller's Entra field with the value the deployment uses
A caller-sent Entra field skipped the cooldown only when it matched no
configured source, so a value equal to an environment default the deployment
overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own
and let a caller's rejected credential cool down a working deployment.
The check now resolves the value the deployment would use without the caller,
in order from the router defaults, the deployment, its named credential, the
AZURE_* environment variables and the default scope, and skips the cooldown
only when the caller's value differs from it.
* fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged
* fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider
* fix(azure): keep the streaming Entra rejection inside the failed attempt
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): check Anthropic-format request tools in tool_permission pre_call
The pre_call hook only collected tools shaped as {"type": "function", "function": {...}},
so flat tools from /v1/messages ({"name", "input_schema"}) and /v1/responses
({"type": "function", "name"}) were never checked and deny rules silently passed.
Reuse the shared anthropic_tool_names helper so every request tool name is checked,
and handle Anthropic tool_choice {"type": "tool"} in both block and rewrite modes.
Fixes#40583
* fix(guardrails): detect Responses API named tool_choice in tool_permission
Responses API named choices put the name at the top level ({"type": "function", "name": ...}),
so rewrite mode removed the denied tool but left a tool_choice pointing at it.
* test(guardrails): run tool_permission pre_call tests through real default_on gating
Build the guardrail as an always-on pre_call guardrail instead of patching should_run_guardrail.
* test(guardrails): type the new tool_permission pre_call test parameters
* fix(guardrails): keep the Responses API custom tool type distinct
Anthropic's user-defined tools carry type "custom", so pre_call mapped
that to "function" for tool_type rules. The Responses API has its own
custom tool type, and mapping it let a custom tool slip past a
tool_type deny rule when the default action is allow.
Normalize only for the formats where custom means a plain function, and
read tool names directly in the rewrite path, which never needed a type.
* test(guardrails): cover flat tools through Messages and Responses
Add blocked and permitted flat-tool cases through the running proxy, with
endpoint coverage and metadata for the current e2e harness
Require a guardrail policy verdict when checking denied tools so an
unrelated provider error cannot satisfy the regression test
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
* fix(guardrails): check namespace and nameless tools and relax tool_choice in tool permission rewrite
* fix(guardrails): check forced, grouped, loaded, and hosted tools by their real type
* fix(guardrails): drop references to removed tools and count loaded tools in tool permission rewrite
* fix(guardrails): check tools sent outside top-level tools in tool permission
* fix(guardrails): judge namespace members by their bare name as a fallback and keep rewrites valid for Gemini, Bedrock and hosted tools
A namespace member is checked by namespace__name first, and by its bare name when no rule names the joined one. Rewrite mode now prunes Gemini allowedFunctionNames and turns off forced calling when no function is left, drops a removed hosted tool from allowed_tools, reads Bedrock toolSpec and systemTool blocks by name, skips cachePoint markers, and leaves a tools field that is not a list for the provider to reject instead of failing with a 500
* fix(guardrails): let only name rules decide a namespace member's bare-name fallback
* fix(guardrails): judge malformed tool types, empty Gemini name lists and pass-through Anthropic tools like the endpoints do
A tool or tool_choice whose type is not a string now reads as untyped instead of crashing the guardrail with a 500, and a null type key is never read as a Gemini tool part. An empty allowedFunctionNames list no longer turns off the functions a rewrite kept. An Anthropic custom tool reads as a function by its input_schema on every route, so a pass-through route judges it the same as /v1/messages. A Responses WebSocket frame now forwards the extra_body tools a rewrite stripped
* fix(guardrails): judge a named hosted tool by its type and keep a nested chat custom tool custom
* fix(guardrails): keep judging Anthropic client tools and type-named server tools by their name
* fix(guardrails): judge a named tool by its name alone only when the name is its type's own
* fix(guardrails): judge the function a tool names beside a declaration list or a top-level name
* fix(guardrails): judge Gemini hosted keys whatever the tool type and a namespace's function
* fix(guardrails): judge legacy functions entries like tools and send Gemini a namespace's members
* fix(guardrails): judge a namespace member by every name a provider reads and send Gemini its schema converted
* fix(guardrails): judge a legacy functions entry as a function and relax a tool_choice in extra_body
* fix(gemini): keep a namespace whole when inlining its members would repeat a function name
* fix(guardrails): judge a namespace member by every bare name that shares its qualified name
* fix(vertex_ai): declare a repeated namespace member name once instead of sending namespace names to Gemini
* fix(guardrails): judge a tool_choice sent in extra_body like a top-level one
A forced name only extra_body.tool_choice carried was never checked, so block mode let a denied name through to the provider
* fix(guardrails): judge a Gemini namespace member returned by its bare name and force a named namespace's members
* test: keep formatter churn out of the tool_permission and Gemini test files
* fix(guardrails): judge extra_body.input tools, argument-aware alias rules, and exact forced Gemini functions
Tools loaded in extra_body.input are now checked and stripped like top-level input. A returned
namespace member is judged by the alias whose first name rule its arguments satisfy. A forced
Gemini name that matches a top-level function keeps that function. Alias grouping and Gemini
declaration dedupe now run in one pass instead of quadratic scans.
* fix(guardrails): judge and rewrite every tool field sent inside extra_body
extra_body is merged into the upstream body, so functions, function_call,
mcp_servers and toolConfig nested there reached the model unjudged. Treat
extra_body as a second request layer that is collected and rewritten the
same way as the top-level request.
* fix(guardrails): judge Gemini hosted keys by key and count only the tools a provider receives after extra_body
* fix(guardrails): relax a tool_choice that names a dropped namespace member by its bare name
Gemini and Anthropic chat send namespace members under their bare names, so rewrite mode now treats a dropped member's bare name, and an emptied namespace's own name, as removed when no kept tool still carries it
* fix(gemini,guardrails): keep a top-level function over a same-named namespace member and leave malformed tool values to the provider
Gemini and Vertex chat now declare a top-level function over a namespace member that shares its name, so a forced call gets the top-level schema. post_call judges a returned name a top-level tool carries as that tool, and pre_call no longer raises on an unhashable or non-string tool name, input item type, or function_call name.
* fix(guardrails): judge a returned namespace member named like a hosted MCP server as that member
* fix(gemini,guardrails): send a namespace as main did when another tool has a member's name
Gemini chat inlines a namespace's members only when no other tool, namespace
member, or legacy functions entry declares the same name, and otherwise sends the
namespace by its own name as main did. A tool_choice naming such a namespace
forces the namespace itself. The tool permission post_call reads top-level names
the way Gemini declares them and judges a call the Responses bridge tags with a
namespace as that member.
* fix(gemini): send colliding namespace members as namespace__name functions and judge them in post_call
When a namespace member shares a name with another tool, Gemini chat now gets that namespace's members as namespace__name functions instead of one function named after the namespace, so every function it can call is a name the rules judged. Declared names are counted once, and a forced member maps to each holder's qualified name.
post_call judges a returned namespace__name as that member, by its qualified or bare name the way pre_call does, so a bare-name allowlist or denylist applies to it.
* fix(gemini): keep generated member names unique and let a top-level function own its exact name
A namespace that listed a member twice, or a generated namespace__name that
repeated a top-level function, sent Gemini two declarations with one name, and
Gemini answered 400 where main answered 200. A name a namespace lists twice is
now one member, and a top-level function keeps its exact name.
post_call now judges a returned name a top-level function declares as that
function, never as the namespace member the name could also mean.
* fix(gemini): qualify namespace members by the namespace's top-level name
The guardrail judges a member as `name__member` from the namespace's top-level
name. Gemini built the name from `function.name` when the tool also carried one,
so a request could pass pre-call and reach Gemini under a name no rule judged.
* fix(vertex_ai): force only the namespace members Gemini is sent
A forced namespace or member on Gemini chat could still allow a generated name that a top-level function or an earlier namespace owns, so tool_choice forced an unrelated function. The forced names now come from the same collision checks the declarations use
* fix(vertex_ai): reject a forced namespace Gemini gets no member of, and resolve forced names in one pass
* fix(guardrails): judge a namespace member by a rule on its namespace's name
---------
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed
A custom video provider can report a video's cost on the status poll that
sees it completed. Every poll of a video shared the video id as its spend
log request_id, so the paid poll's row was dropped as a duplicate and every
later poll of the finished video charged the cost again.
Video status rows now carry their own `_video_cost` request_id, and a poll
that reports a cost goes through the same settle-once claim batches use, so
the first poll that sees the video completed charges it and writes the row,
and every later poll charges nothing.
* fix(proxy): charge a video's provider-reported cost once on model, router and session budgets too
A video status poll gets the _video_cost row suffix only when it reports a cost, so a $0 poll keeps the bare video id and adds no row of its own, and the spend log writer claims a row by that suffix.
The batch charge-once marker now covers videos, keyed on the video id under {spend_key}:video:{id}, with the batch marker key unchanged. Model and router budgets use it, and per-session budgets now charge a repolled batch or video once and ignore a cost that is not a finite number
* fix(proxy): charge a finished video's own cost once, keyed by the id the client polled
Guardrail cost on a running poll no longer takes the charge-once claim, only a completed poll charges, the session marker TTL refreshes like the others, and two deployments whose provider ids match each keep their own charge.
* fix(proxy): charge repeat poll guardrails, key video charges by canonical id and survive redaction
* fix(proxy): keep video status polls in-process for the collector and give guardrail-only job polls an unclaimable row id
* fix(proxy): keep repeat batch polls free and group guardrail-only batch rows under their batch's trace
* fix(proxy): settle a finished job whose own cost is below its poll's guardrail cost under the job's charge row
* test(proxy): count charged_poll_cost as covered through the finished job guardrail poll tests
* fix(proxy): settle a finished batch that cost nothing itself on its first priced retrieve
* fix(proxy): charge no budget for a batch retrieve before its cost is final, and read the retrieved batch from before redaction
* fix(proxy): charge sessions a poll's guardrail-inclusive cost, count a job cost row's guardrail run, and keep batch_cost_is_final importable from litellm.batches
* fix(proxy): claim a job and charge the session budget in one Redis call
The session limiter claimed the job marker and then raised spend in a second Redis call. A failure between the two left the job claimed while its charge sat only in local memory, so later polls skipped it. One script now does both, and a refused script claims and charges locally
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: update EmpirioLabs models in model_prices_and_context_window.json
* chore(endpoints): regenerate provider endpoint support for the EmpirioLabs cost map rows
* fix(empiriolabs): drop the retired max_tokens field, route embeddings and remove the unrouted rerank row
The cost map test rejects any row carrying max_tokens, so every EmpirioLabs row keeps max_input_tokens and max_output_tokens only. The provider JSON now lists /v1/embeddings so embedding(model="empiriolabs/text-embedding-v4") reaches the OpenAI-compatible embeddings route instead of failing provider lookup. The qwen3-rerank row goes away because no rerank route exists for the provider. Endpoint support regenerated from the cost map
* fix(empiriolabs): mark audio input and image embedding input on the rows the provider lists as multimodal
The three vision embedding rows get supports_embedding_image_input and the eleven chat rows whose input_modalities on GET https://api.empiriolabs.ai/v1/models include audio get supports_audio_input, so capability checks stop treating them as text only
* fix(empiriolabs): carry the long-context price tiers on the 24 rows the provider bills in bands
EmpirioLabs bills 24 of these models at a higher rate once the prompt passes 32K, 128K, 256K, 272K or 512K tokens, and the rows carried only the first band, so a 300K-token qwen3-7-plus request was priced at a third of the real cost. Each of those rows now carries input_cost_per_token_above_<N>k_tokens, output_cost_per_token_above_<N>k_tokens and, where the row already has cache_read_input_token_cost, cache_read_input_token_cost_above_<N>k_tokens for every band, taken from pricing[].min_context on GET https://api.empiriolabs.ai/v1/models for the base rows and from the Variants rate card on docs.empiriolabs.ai/models/<id> for the six variant rows
* feat(empiriolabs): bill the per-call web search fee reported in usage.tool_usage
EmpirioLabs charges a flat fee per invoked web_search, web_search_image or image_search call and reports the counts in usage.tool_usage on the chat response. The JSON provider config now carries that count as usage.server_tool_use.web_search_requests when providers.json sets special_handling.web_search_requests_from_tool_usage, the per-call web search cost path bills the usage-reported count instead of flooring to one call, and the 39 EmpirioLabs rows the provider lists a per-call search price for carry it as search_context_cost_per_query, from unit_pricing on GET /v1/models for base rows and the Variants rate card on docs.empiriolabs.ai for variant rows
* fix(openai_like): mark the dict signature of transform_parsed_response_dict as matching BaseConfig for LIT001
* fix(openai_like): mark both dict annotations of transform_parsed_response_dict for LIT001
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Moves the page-authored English on /playground, the /chat app and /moyai
into t() and Trans with keys in en.json, as a pure text move. The rendered
English is unchanged
Route the page-authored English on the Logs, Usage and Telemetry pages,
the log detail drawer and the shared usage widgets through t() and Trans,
with 1,250 messages added to en.json under 10 new top-level keys. The
rendered English is unchanged.
Moves the page-authored English on the virtual keys, projects, cost tracking,
cost optimization and budgets pages, plus the create key form and the shared
key and team pickers, into t() and Trans with 1,318 messages under 32 new
top-level keys in en.json. The English on screen is unchanged
key_edit_view.tsx grows past 800 lines, so it gets a max-lines entry in
eslint-suppressions.json instead of a split
* fix(router): stamp the failed deployment on every endpoint's attempt
Weighted failover (#27980) re-picks within a model group by excluding the
deployment named in the error's failed_deployment_id. #27980 stamped that id
in only three attempt functions: _completion, _acompletion and
_ageneric_api_call_with_fallbacks_helper. Embedding, image generation,
transcription, speech, text completion, rerank, adapter completion, the sync
generic call and vector store create never stamped it, so with
enable_weighted_failover on, a failure on one of their deployments skipped the
in-group re-pick and went straight to the cross-group fallbacks, or raised.
Each of those attempts now stamps the deployment it picked, read from the
model_info that _update_kwargs_with_deployment put on the request (so a
client-side-credential call names its own dynamic deployment, as on
completions). A failed pick stamps nothing. File and batch creation and batch
cancellation are left alone: they are bound to a resource on the deployment
they picked, so re-picking another one cannot help.
* test(router): narrow the failed-pick raise to the no-healthy-deployments error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013FDwtoMjnHa4xxazGJJNSm
* fix(router): exclude the configured deployment on failover for client-side credentials
Weighted failover excluded the per-request id that client-supplied credentials
generate, so the failed configured deployment stayed eligible for the re-pick.
Stamp the configured id alongside it and exclude that. Also drop the vector
store create stamp, which no fallback path reads, and redundant test comments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013FDwtoMjnHa4xxazGJJNSm
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(decisions): add EmpirioLabs AI as a decisions provider
empiriolabs/aplomb-1 now works with litellm.decisions and litellm.systemone, the proxy /v1/decisions and /v1/systemone routes, the decision model guardrail and the Admin UI. Requests go to POST /v1/decisions on the EmpirioLabs AI API with the System One body, keyed by EMPIRIOLABS_API_KEY with an optional EMPIRIOLABS_API_BASE
Chat models keep routing through the existing OpenAI-compatible JSON provider, and a test pins that
* ci: run the EmpirioLabs AI decisions tests in the llms unit shard
* fix: align EmpirioLabs registration with CI gates
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
