* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust
* fix(model-cost): keep main's compat helpers pure and move network tests to integration
ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): add litellm-s3 crate, AwsCredentialsProvider and cost map fetch
* fix(model-catalog-refresh): time each read instead of the whole cost map download
The 5 s fetch timeout covered the full body, so a slow but steady link failed every attempt where httpx and boto3 would finish. Each read now gets its own 5 s, matching main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Google AI Studio answers generationConfig.candidate_count "" with a 400 before it generates anything, so the truncated 200 this case scripts is a reply Google never gives. The other malformed count and cap cases stay.
* fix(router): let a retry leave a pinned deployment after a retryable failure
The retry loop only recorded a failed deployment for non-retryable statuses,
so encrypted_content_affinity re-pinned the failing origin on every 5xx, 429
or connection-error retry. Track every deployment the request failed on and
hand that list to the callback filters, leaving selection's skip list alone.
Fixes#47607
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): carry failed deployments through stream retries too
retry_stream started each retry with no failed ids, so an affinity pin on a
stream that failed early stayed on the deployment it failed on. Seed the list
with the stream's deployment and add each retry that fails. Drop the 429
python case, whose RateLimitError mock cannot be deep-copied by the router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(router): cargo fmt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): leave out deployments that failed the request when affinity strips
When the origin had failed this request and no boundary peer existed, the
affinity check stripped the reasoning and returned every candidate, the failed
origin included. Later retries then carried no marker and re-picked the failed
deployment one time in three. Return the untried candidates instead.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-messages): apply additional_drop_params before the unsupported-param gates
* fix(anthropic-messages): apply additional_drop_params before the unsupported param checks
Share the tolerant drop list parser between the Python handler and the Rust host route, clear listed params before the Rust gate, and keep the prompt cache baseline body in step with the real request
* fix(anthropic-messages): leave additional_drop_params keys out of the Rust host body
* test(anthropic-messages): annotate the new test locals with Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): merge default metadata under the request's instead of replacing it
* fix(router): join default metadata tags after the request's tags
A router default_litellm_params.metadata.tags list replaced the request's tags, so the deployment and credential tags the router adds never reached the logged request tags. Default tags now follow the request's tags, skipping ones already present, like the proxy's key tag merge
* fix(router): keep masking a router default turns on
Merging default metadata under the request let a caller's mask_input or mask_output false switch off masking an admin set in default_litellm_params.metadata. A masking flag the default turns on now stays on, and a caller can still turn masking on. Also gives this PR's test suppressions a reason and adds a missing blank line
* fix(router): keep the router's cache namespace and drop malformed caller tags and dicts
A caller's metadata.redis_namespace no longer beats a router default, so it
cannot pick another tenant's cache partition. A caller tags value that is not a
list, or a non-dict value under a dict default, is treated as absent the way the
proxy's merge_tags treats malformed tags, so the router defaults still apply.
* test(router): check that mask_output stays on when the router default turns it on
* fix(router): keep a router default guardrail whatever guardrails the request lists
* fix(router): keep the admin's settings for a router default guardrail when the caller lists it
* test(router): check that a router default guardrail entry keeps the admin's settings
* fix(router): keep a router default private Langfuse trace private whatever the caller sends
* fix(router): keep a router default function such as langfuse_masking_function over the caller's value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Several integration tests on main still assert behavior that later changes replaced on purpose, so they fail on every scheduled run, and one of them stops the whole sdk group at import.
The batch cost case expected cached tokens to bill at the Batch input price when a price row has no Batch cache price. Since #50062 they bill at the Standard cache price, so the case now expects 0.000235 input cost and is renamed to say so.
The OTel v2 attribute test pinned langfuse_otel and langtrace as carrying no request metadata baggage. Since #50373 every OTel v2 destination carries it, so all seven callbacks share one expectation and the Langfuse baggage test that was deferred under LIT-9128 runs again.
The clean-EOF outage test trips its database relay on the text of the spend log insert. The ledger writes that insert without the "public" schema prefix, so the relay never tripped and the test timed out before it checked anything. The trigger now matches the insert on both write paths.
The usage routing tests expected cooldown, tpm and rpm keys in one MGET. Since #50170 the cooldown keys are read in their own MGET, at most once per DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS. Both the proxy and the SDK test now expect the two MGETs, and the proxy test waits out that interval after its /model/info poll, which reads the same keys.
The concurrent deployment delete test held its writer inside Router._update_deployment_indices_after_removal, which went away with the Python router core in #50178, so the file failed at import and stopped the whole sdk group. It now holds the writer where the registry reads the model list again after router-core applied the change.
Python stores a unary messages response from a background cache write task that runs after the router stamps _hidden_params, so the entry carries model_id and the router headers and a later hit returns x-litellm-cache-key. The native path stored the entry inside the native call, before the router saw the response. The async unary store is now deferred until the route encodes its public response and is handed to Python's own cache write task
When the client disconnects mid-stream, the native adapter now logs success for the chunks it delivered before teardown, so spend, SpendLogs and the key's budget and tpm reservations settle the same way they do on the Python path
The proxy records a client disconnect, with its 499 error information, in the stream generator's cleanup. The native teardown billed the stream before that cleanup, so the row had empty error information. A proxy stream torn down by cancellation now bills from the task's done callback, after the disconnect is recorded
A stream whose upstream body ends mid-frame reached proxy callers as a mapped APIConnectionError with status 500. Python's transport raises an unmapped httpx error at that point. The native body read now records a body cut and the host raises the same exception the active transport would
Python wraps a connect, DNS, TLS or reset failure in a synthesized 500 AnthropicError, which maps to InternalServerError. The native path surfaced these as APIConnectionError with the bug-report footer, which also changed cooldown decisions
The proxy bills the estimated prompt of a failed call only when the Timeout was caused by an httpx ReadTimeout. The native timeout now chains the same httpx timeout Python's client raises, so its failure row bills the prompt
Python rebuilds an upstream error response from the decoded body without content-encoding and content-length. The native failure now drops those wire headers, and a failure raised without an HTTP response states no headers at all, like Python
Python's async client wrapper stamps the effective timeout and num_retries on every failure, which the proxy reads for x-litellm-timeout and the router reads for retries. The native async paths skipped that wrapper, so error responses from Rust-served async calls had no x-litellm-timeout header
Python's async chat completions, responses, embeddings and messages entries hand the call to run_in_executor before they build the provider body, so a during_call guardrail started next to the call masks the request first. The native paths built their request without yielding, so masked text never reached the provider
The shared native_acall_hook now yields once and stamps the failure, and every async dispatch passes it as its native hook. OCR passes native_acall_hook_on_loop, which only stamps the failure, since its Python entry went through the client wrapper but built its request on the loop
Python's messages transform takes any JSON object from a 200 response and only adds stop_sequence, so a provider answering 200 with an error object reaches the client as 200 and the router counts a success. The native decoder turned it into a 500 APIConnectionError and a router failure. A JSON object that is not a message now passes through unchanged
A non-string extra_headers or deployment header value surfaced from the native handler as an APIConnectionError. It is now refused as an invalid request when the headers are collected, before the provider request is built or logged, so the caller gets a 400 BadRequestError naming the header and its type
Python logs the requested optional params, taken before the provider transform, as model_parameters. The native handler logged the transformed body, so a reasoning_effort request was logged as thinking and output_config. The request context now carries the requested params
Python updates model_call_details with the transformed request body before pre_call, so the logged messages are what was sent. The native adapter kept the caller's messages and compared params as JSON values, so an int 0 logged as 0.0
Python only records custom_llm_provider in model_call_details. The native adapter also wrote it into litellm_params, and the proxy reuses one logger across router hops, so a fallback hop's pre_call log showed the provider where Python shows null
Python only stamps cache_hit when a result came from the cache. The native adapter wrote False on every provider result
Python calls pre_call with the request body as a json user message, and a stream's spend-log cache_key is computed over that input. The native adapter logged the string Messages, so a streamed call's cache_key differed
* fix(messages): log native messages success once, after the proxy releases it
The proxy defers non-stream success logging past post-call guardrails, but the native host parked its log in its own slot and the release only ran for OCR, so a unary /v1/messages call served by Rust logged no success event, no SpendLogs row and no key, team, user or end-user spend. A deferred native success now parks in enqueue_deferred_logging with deferred_success_result, the same slots Python uses, so the proxy releases it like any other call and its guardrail step pruning and cost preparation see it too
Python runs sync success callbacks inside the deferred logging closure, so a post-call hook that blocks the response drops them. A parked native success now carries its sync callbacks and runs them on release
A refused first attempt and its fallback share one logger. Python logs the async success once, for the first attempt, so the native adapter now claims the async success log once across fallback attempts. Like Python, an attempt that loses the claim runs its sync callbacks only when logging is not deferred, and an internal call logs nothing
* test(messages): cover the deferred native success park and release
The proxy attaches its shared aiohttp session to every request and a default stream_timeout to every stream, and the messages detectors treated both as Python-only features, so no proxy request reached Rust. A shared session is only a connection-reuse hint and no native client reads it, so the common caller_client detector now fires only for an explicit client. The native messages call now takes the one timeout Python's messages handler resolves from timeout, request_timeout, stream_timeout and an httpx.Timeout, so no timeout shape keeps a messages call on Python
* feat(proxy): let a key's logging entry reference a proxy credential
A virtual key's metadata.logging entry can name a proxy credential with
litellm_credential_name. At request time the credential's values, which config
credential_list resolves through the secret manager at boot, are merged under the
entry's callback_vars, with the entry's own values winning. An entry whose
credential is not defined is dropped, so it never falls back to the proxy's own
callback environment.
Only proxy admins may change the logging entries of a key when the stored or
requested list holds a reference (403), and a reference to an undefined
credential is a 400. The check runs on key generate, update, team bulk update,
regenerate and /user/new, and compares decrypted entries so re-saving metadata
read back from /key/info is not a change.
Resolves LIT-10061
* fix(proxy): drop a key logging entry whose referenced credential holds no logging values
* fix(proxy): only require newly added credential references to exist on key writes
* fix(proxy): log a dropped credential reference as an error so /key/health reports it
* fix(proxy): gate credential references on bulk-created user keys
* fix(proxy): keep failing requests on keys with non-object logging metadata
A key whose stored metadata.logging is not a list of objects failed every
request with a 500 before this branch. The credential reference rewrite
turned those values into no entries, so the request quietly succeeded
without the key's logging. Raise a TypeError for those shapes again so the
request path behaves as it did on main, while empty values and objects with
unknown fields still go through the existing skip.
* test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging
The malformed logging cell expects the base behavior that 7ac4ab1356
restored, a 500 for a key whose stored metadata.logging is not a list of
objects and a 200 for an empty list
* fix(proxy): keep a 500 for teams with non-object logging metadata
On main, a team whose stored metadata.logging held non-object items failed
chat requests with a 500, because AddTeamCallback(**item) raised a TypeError.
This branch switched to AddTeamCallback.model_validate, whose before-validator
raises AttributeError on those items, and the proxy maps AttributeError to a
400 invalid_request_error. Raise a TypeError for non-mapping items before
validation so the status matches main again, as the key path already does.
* fix(complexity_router): keep Claude Code subagent and side-request traffic off other agents' tier pins
Claude Code sends one x-claude-code-session-id for the main agent, every
subagent and their side requests. In user_turn mode a subagent's new ask
rewrote the parent's pin (#50099), and the agent progress summaries Claude
Code fires during a subagent run rewrote the subagent's pin to the cheapest
tier (#50444).
Subagent requests now get their own tier pin keyed on session id plus
x-claude-code-agent-id. Requests marked x-claude-code-request-class:
auxiliary route as housekeeping, and only main and subagent turns may write
a pin. Without the hint header, the summary and prompt-suggestion prompts
are matched as housekeeping sentinels.
* chore(ui): regenerate schema.d.ts for the housekeeping_patterns description
* fix(complexity_router): let the request-class hint only mark housekeeping, not suppress pin writes
A client-sent request class must not decide whether a decision is saved.
Tagging a real ask as workflow or auxiliary dropped its routing decision,
so later continuations replayed an older pin with the new history. The
auxiliary hint now only routes the request through the housekeeping path,
which is already never pinned and defers to custom classifiers and
keyword rules.
* test(complexity_router): cover the auxiliary hint on a prompt no sentinel matches
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): read an absent team membership row as zero spend under fail-closed budget enforcement
A team user under a default member budget has no LiteLLM_TeamMembership row until spend is persisted. SpendCounterReseed.from_db returned None for that absent row, the same value as a failed DB read, so fail_closed_budget_enforcement rejected every request from that user with 503. The DB answering that the row is absent is a verified zero, so from_db now returns 0.0 for a team-member counter without a row and still returns None when the read fails.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only read an absent membership row as zero when the team member key parses unambiguously
A team_id or user_id containing ":" makes the counter key split onto the wrong membership row, so that miss keeps returning None instead of a verified zero. Type the new unit tests.
* fix(proxy): refuse a fail-closed team member with no membership row with a 403 that names /team/member_add
Replaces the earlier reading of an absent LiteLLM_TeamMembership row as verified zero spend. A key minted with team_id and user_id but no /team/member_add has no row and the spend writer never creates one, so admitting it at zero would leave its member spend tracked only in Redis. With fail_closed_budget_enforcement on, _check_team_member_budget now rejects that key with a 403 that names the user, the team and the member_add call, before any spend read. With the flag off nothing changes. The membership load raises on a database failure and returns None only for a genuinely absent row, so the 403 never fires during an outage. spend_counter_reseed.py and the counter prefix constant go back to main.
* test(proxy): drive the no-membership 403 tests through the real membership and spend readers
Fake Postgres rows and the FakeSpendCounters double stand in for the external stores instead of patching get_team_membership and get_current_spend. Strip CR/LF from the ids in the fail-closed warning log.
* fix(proxy): read fail_closed_budget_enforcement from general_settings in the membership check
Importing proxy_server's private helper grew reportPrivateUsage past the type gate ceiling. Type the integration helper's return as JsonValue.
* fix(proxy): expose fail_closed_budget_enforcement_enabled for the membership check
Reading general_settings directly from auth_checks grew the unknown-type counts past the type gate ceiling; a small public wrapper in proxy_server keeps the gate flat.
* test(proxy): drop docstrings that restate the no-membership tests
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover claude-fable-5-2 on Vertex AI and Bedrock Converse
Scripted-upstream wire tests for the Fable capability flags on both routes, the
vertex_ai and us.anthropic cost cases every other Fable row has, Fable 5.2 in
BEDROCK_CONVERSE_MODELS, and bedrock/claude-fable-latest repointed to 5.2 like
the Anthropic and Vertex aliases already are.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): add GovCloud Bedrock rows for claude-fable-5-2
bedrock/us-gov-west-1 and bedrock/us-gov-east-1 rows copy the Fable 5.1 field
set with the GovCloud rates from the Bedrock price list.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): let deployments declare a shared encrypted content boundary
Deployments that set the same model_info.encrypted_content_boundary are
treated as encryption boundary peers by encrypted_content_affinity, so
regional endpoints of one provider organization with different api_base
values keep encrypted reasoning when a follow-up moves between them.
Deployments without the key keep the api_base and api_key match.
* refactor(router): type the encryption boundary peer lookup's deployments
* test(router): cover order fallback hops onto a declared encryption boundary peer
* chore(ui): regenerate API types for encrypted_content_boundary
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(bedrock): answer 400, not 500, for malformed native count-tokens requests
The Bedrock count-tokens handler now checks the body before any AWS call and raises a 400 BedrockError for a missing or empty messages list, a message without role or content, or a missing model. The token counter marks that rejection as local, so it skips the Mantle fallback and the framing estimate, and /v1/messages/count_tokens answers invalid_request_error
get_bedrock_base_model returns a bare cross-region prefix such as us or global unchanged instead of raising IndexError, so Bedrock judges the model id and its own 400 reaches the caller
* fix(bedrock): answer 400 for a malformed message on Mantle token counts
* fix(bedrock): limit the region-only model id fix to token counting
* fix(bedrock): count a routing-prefixed region-only id and leave other providers' count errors as they were
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
python-bridge now installs ring in its pymodule_init, matching the gateway's
startup install. redis-connect no longer sets the process default as a side
effect of connecting, so a dependency that calls ClientConfig::builder()
without a provider cannot panic on the Python host depending on whether Redis
has connected yet
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Rust credential source passed gcp_service_account straight into the
IAM credentials URL, so a bare email got a 404 and the stores fell back
to per-worker memory. Address it as projects/-/serviceAccounts/<email>
when it is not already a resource name, like the Python gRPC path did.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ollama_chat): give each streamed tool call its own index
Ollama streams parallel tool calls one per chunk with no top-level index, and the delta
numbers the calls of each chunk from zero, so every call left the proxy as index 0 and
OpenAI-style clients appended the arguments of each later call onto the first one. Number
the calls with a counter that lives for the whole stream instead.
Fixes#33678
* fix(ollama_chat): answer 400 when replayed tool call arguments are not JSON
A chat history that already holds a merged tool call from an older stream
used to fail with a 500 APIConnectionError. It now answers 400 naming the
tool call. The streaming tests use Ollama's real tool call shape and drop
a case the merge base already passed.
* fix(ollama_chat): parse replayed tool call arguments with the shared repair parser
Empty, null, and truncated arguments in a replayed assistant tool call now go through parse_tool_call_arguments, so they are forwarded as an object instead of answering 500. Arguments that still are not a JSON object answer 400 naming the call id.
* fix(ollama_chat): send {} for null replayed arguments and keep argument text out of the repair warning
Replayed tool call arguments of the JSON string "null" used to reach Ollama and return 200. The shared parser turns them into None, so they now go out as {} like empty or missing arguments instead of answering 400.
The parser's repair warning logged up to 200 characters of the arguments. It now logs only the tool name, context, and length.
* fix(ollama): keep replayed tool arguments out of a rejected replay's error chain
* fix(ollama): keep the repair warning's argument preview unless message logging is off
The shared repair warning lost its 200-character argument preview for every provider. It now logs the same preview as before by default, and logs redacted-by-litellm instead when turn_off_message_logging is on.
* test(ollama): cover short and boundary-length repair warning previews
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(minimax): forward reasoning_effort instead of dropping it
* fix(minimax): forward reasoning_effort instead of dropping it
* fix(minimax): gate reasoning_effort on the cost map and map MiniMax-M3.1-Flash
Forward reasoning_effort for every MiniMax model unless its cost map entry says it
does not reason, so a MiniMax model newer than the map keeps the caller's effort
instead of losing it.
Add a minimax/MiniMax-M3.1-Flash entry with MiniMax's published pay-as-you-go
pricing and its accepted effort levels (low, medium, high, xhigh, max), and a
minimax-reasoning-baseline fallback rule that marks unmapped minimax/MiniMax-M*
ids as reasoning-capable, so thinking is kept for them too.
* refactor(minimax): gate reasoning_effort on supports_reasoning like thinking
Forward reasoning_effort when the cost map, or the MiniMax M-series fallback rule, says the model supports reasoning, the same check thinking already uses. An unlisted MiniMax model outside the M series now drops it, with the existing drop warning.
* refactor(minimax): build supported params in one expression and type the new tests
Build the MiniMax supported params list without mutating it, give the new test helpers precise types, and assert invariants (both cost map files agree, prices are positive and ordered, every declared effort level is forwarded) instead of pinning MiniMax's prices and effort levels as literals.
* fix(minimax): treat every unlisted minimax/ model as reasoning-capable
Widen the MiniMax fallback rule from M-series ids to the whole minimax/ namespace, matching the W&B rule, so an unlisted MiniMax model keeps reasoning_effort and thinking. Exact cost map entries still win, so mapped speech, video and image models are unaffected.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(auth): grant access groups through principal attribute rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): satisfy cache tracing and recursion CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): close access group rule authorization bypasses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): reject self-editable identity rule attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Centralize JWT model authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Cover centralized key checks and name denied JWT models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Test centralized authorization for custom auth and JWT mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve model normalization before budget validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use a controlled clock for WebSocket session deadline tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Unify JWT credential admission and simplify team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Simplify request-time access group authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use real requests in custom auth budget fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When the proxy parks the success log of a non-streaming response and the
router then discards that response (a safeguard refusal it falls back
from), the parked log was sent in place of the serving attempt's, so the
spend log and cost header reported only the refused attempt and the
serving attempt was never billed.
Carry the parked response as a billed attempt owed by whichever attempt
ends the request, the same way a superseded stream is already billed, so
one spend row and the cost header cover every billed attempt.
Fixes#49103
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* test(e2e): cover claude-fable-5-2 capability flags and prompt-cache pricing over Anthropic direct
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): replace the fable-5-2 e2e suite with scripted-upstream wire and cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add claude-fable-5-2 wire and cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): add Claude Fable 5.2 on the Anthropic API, Vertex AI and Claude Platform on AWS
Adds claude-fable-5-2, vertex_ai/claude-fable-5-2 (plus @default) and
bedrock/claude_platform/claude-fable-5-2 at $8/$40 per MTok, $10 and $16
cache writes, $0.20 cache reads, 50% batch discount and a 2x fast mode
multiplier, and points claude-fable-latest and vertex_ai/claude-fable-latest
at the new rows. Bedrock converse rows ship separately in #50707.
* fix(cost-map): drop fast mode from the Vertex Fable 5.2 rows, the Vertex cost path does not bill it
* Merge main, drop the Anthropic API and Claude Platform rows that #50715 already landed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt_templates): keep moved private helpers importable from common_utils
The split moved 83 private helpers out of common_utils, so code importing them by
their old path got ImportError. A module __getattr__ now serves each one from its
new module. Also renames the new test files to the directory's naming convention
* fix(prompt_templates): hide the old-path getattr from type checkers and serve only released names
Type checkers stop seeing the module __getattr__, so a typo'd import from common_utils
fails lint again. The old path now serves only the 69 moved helpers that common_utils
had in a tagged release, since the other 14 were never importable from it
* fix(prompt_templates): let patches through the old common_utils path reach moved helpers and warn on old-path use
* fix(prompt_templates): sync old-path patches of re-exported helpers and restore moved helpers when warnings are errors
* fix(prompt_templates): reset a moved helper on old-path delete instead of removing it, and warn on old-path writes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* build(ui): fail lint on hardcoded English in dashboard JSX
Adds eslint-plugin-i18next no-literal-string in jsx-only mode with type info for dashboard source files, skipping tests and the two untranslated files, plus a no-restricted-syntax selector that rejects an English literal as the first argument to toast.success, error, warning or info.
* test(ui): lint the fetch rule fixture at a real source path
Type-aware linting needs the file to be in the TypeScript project, so a virtual path under src now fails to parse and no rule runs. The fixture now uses src/app/layout.tsx with the same code and assertions.
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL
* fix(prometheus): carry live spend for the user, org and customer budget gauges too
The remaining budget gauges for users, orgs and customers read the same cached rows as the key and
team gauges, so they read high for about a minute after a charge. Auth now carries the live counter
for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded.
The x-litellm-key-spend header and logging spend fields keep their current values.
* fix(prometheus): show committed spend in the remaining budget gauges
Reads twin counters that take only charged cost, so a worst-case budget hold
never shows as spent and a request that finishes last cannot set a gauge back.
The twins exist only while the Prometheus logger is registered, and every path
that resets a spend counter drops them. Failed requests now set the org gauge
from the carried org row and committed spend, not the cached org row.
* fix(prometheus): count a charge the cached customer row already holds once
* fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets
Auth now reads committed spend through the admission MGET and names the counters a charge
may move, so the master key and unbudgeted entities get no twin. Organization spend edits
and the global spend reset drop their twins, and a long request no longer reads as charged.
* fix(prometheus): read the committed org spend on failed requests without the auth object
* test(prometheus): cover a failed request with no carried org
* fix(prometheus): move the committed counters for charges with no auth on the row
A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them.
* fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes
Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a
charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed
counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys
and teams in bounded pages.
* fix(prometheus): count a charged failure's own cost in the org gauge
A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request
* fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog
* fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete
Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins
* fix(prometheus): keep committed spend values out of logger dumps of pass-through rows
CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values
* fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster
The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files
* fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): add Claude Fable 5.2 and Mythos 5.2 offer index prices
* fix(cost-map): absorb verified prices from #50559
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): remove unsupported Gemini cache-write rates from #50504
Co-authored-by: Chester <42577861+superdiaodiao@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): remove retired Friendli models from #50692
Co-authored-by: Lee-Si-Yoon <76679207+Lee-Si-Yoon@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the eu and us-gov Mythos 5.2 profiles the base model's native structured output flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the eu and us-gov Mythos 5.2 profiles the base model's native structured output flag
* feat(anthropic): add claude-fable-5-2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): add tool and caching flags to claude-fable-5-2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): carry supports_mid_conversation_system on claude-fable-5-2
* fix(anthropic): declare Fable 5.2 mid-conversation system support
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): deduplicate concurrently added Fable capability
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): resolve merge conflict with main, keep main's claude-fable-5-2 row
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chester <42577861+superdiaodiao@users.noreply.github.com>
Co-authored-by: Lee-Si-Yoon <76679207+Lee-Si-Yoon@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(proxy): build plugin session claims from FIPS provider services in fips_mode
With general_settings.fips_mode on, the per-plugin key comes from the
salt key KBKDF and the Fernet token is built from ssl.RAND_bytes,
AES-128-CBC and hmac.digest. Standard mode is unchanged.
* test(proxy): freeze the clock in the fips_mode plugin claim test
