* fix(proxy): bill an OpenAI pass-through background response once across repeated reads
A finished background response now bills only on the first pass-through call
that sees it finished with usage. The claim is an insert on the new
LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds
across workers and replicas. Every later retrieve, cancel, streamed create, or
resumed stream logs like an unbilled read. With no database or a failed insert
the call still bills.
* fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse
* fix(proxy): claim background responses on failed streams and bound the claim
A resumed stream that fails after response.completed billed its usage through
the partial-usage path without claiming the response id, so a later read billed
it again. That path now claims the id and logs $0 when an earlier read paid.
The claim uses create_many(skip_duplicates=True) so a repeat read no longer
raises a unique violation in Postgres, and it waits at most 2 s before billing,
since the claim now runs before the client gets its response.
* fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids
* test(pass_through): cover a failed stream whose failure logging cannot be scheduled
* fix(proxy): start the background response settlement migration with a lock timeout
* fix(passthrough): leave a failed background stream's charge to its next read
A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The test stores its recording Redis client in litellm's shared client
cache with the default ttl. When earlier tests in the same process have
filled that cache with longer lived clients, the next insert evicts the
recording client, the pin is read through a fresh client, and the test
sees no commands on the wire.
* fix(ui): keep the request log model cell to one model
A session row rendered every model the conversation ever used, comma
joined. A long conversation produced a huge string that the auto-layout
table turned into a very wide Model column, pushing the rest of the
columns off screen.
Render only the model of the row's own call (the newest call in the
session) and show an overflow count. The full model list stays in the
hover popup, including a note when the server capped the list.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(ui): use the shared Badge and an i18n key in the request log model cell
The "+N" overflow marker now uses the shadcn Badge instead of a hand-styled span, and the
"More models not shown" note reads from en.json so the dashboard's i18next/no-literal-string
lint rule passes.
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(proxy): let the OS assign Prisma engine ports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear Prisma connection state after failed startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer
Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525,
first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown
Codex versions still get the catalog cut to 1 MiB, since they drop a larger one.
* fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog
Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those
clients counted as too old for the uncut catalog, their custom reasoning levels and stock
entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does.
* test(proxy): type the Codex catalog byte-limit test
* refactor(proxy): gate the uncut Codex catalog on the 0.166 release line
Read only the major and minor numbers of client_version instead of a shared
semver comparison, and leave the reasoning level and stock entry version
checks as they are on main.
* refactor(proxy): read every Codex client_version with one helper
Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version
parser rejects. The catalog byte limit, the per-model minimum version and the
custom reasoning level checks now share one reader for Codex's format, which
also bounds digit counts so an absurd version can't fail the listing.
* fix: make scheduled spend resets idempotent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover reset roundoff with shared Redis workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: isolate reset workers and type raw adapters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep updated_at moving on zero-spend budget resets
* fix: never leave spend below zero after a budget reset
* fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged
Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time
* fix(proxy): keep a resent budget reset time only once it is due
A save that resends the same budget_duration keeps the stored reset time
only when that time has already passed, so the scheduled sweep still
resets the period it is due for. A row that is not due yet realigns its
reset time on save, the same as before.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(transcription): treat libsndfile's unknown frame count as no duration
* refactor(transcription): inline the unknown frame count check
* fix(transcription): read the frame count once to keep the type gate flat
* test(transcription): annotate the new passthrough test's locals as Final
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(budgets): add per-model tpm and rpm limit maps to budgets and customers
Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more).
/budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth
path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and
end_user_model_rpm_limit so the limiter can enforce them in a follow-up.
* fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes
Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata,
which the limiter reads, so the shared budget allowlist leaves those two fields to customers.
Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy
the type discipline gate. Customer info tests expect the two new budget keys.
* fix(organizations): keep stripping budget_id and flattening nested per-model maps on update
The update path pops every budget field from the organization row again, so a caller's budget_id
never changes the attached budget, and a nested litellm_budget_table still flattens the two maps
into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS.
* fix(types): keep basedpyright and ruff strict counts flat for the new budget fields
Reads the per-model maps off the end-user params with membership checks instead of .get on an
untyped dict, writes the inline customer budget through the client's typed jsonify_object, and
builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never
receive object-typed kwargs.
* chore: drop unrelated reformatting from project endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: update the spend tests that still expected the pre-ledger accounting queues
#49872 made the SpendLogs ledger the default accounting path on purpose. A batch with no billable request now writes no cost row, a call the guardrail fails closed during a vendor outage keeps one spend row of either status while its key is still charged once, and charges left at a graceful shutdown are applied by the next start. The daily spend batch shutdown tests exercise the old queue, so they pin use_spend_logs_ledger to false.
* test(integration): price the guardrail relay model so the outage test can assert the charge
The relay model had no price, so every stream cost $0 and the outage test could not tell a charged call from an uncharged one. Both rigs now price it, and the test expects each served stream to cost 9 input and 4 output tokens and the key to be charged for all sixteen.
Also notes the skipped batch cost row case in the e2e rules file, next to the other explicit skips.
* test: repair four tests that fail for reasons outside the product
The OpenAPI docs gate test rebuilds the Rust extension with uv, and the integration lane blocks outbound traffic, so uv could not refresh its PyPI index for maturin. The test now runs uv offline against the cache the dependency install already filled.
The response cache test expected a chat reply blocked with empty content to be cached. Since #50057 a reply with no output is never cached, so that case now expects the repeat to reach the upstream and the real answer to be cached.
The Gemini cache hit e2e test gave the model 16 tokens, which its thinking can use up, and a reply with no text is not cached. It now allows 512 and checks the first reply has text.
The Together cache read e2e test picks the cheapest model with a cache read price. Since #50491 that is a -latest alias, whose target reports no cached tokens. Aliases are now left out of the pick.
* fix(complexity_router): user_turn continuations no longer replay an earlier ask's pin
A user_turn pin was stored per session, so when a new ask could not be
pinned (classifier timeout, housekeeping excluded) its tool continuations
replayed the previous ask's decision and could silently drop a hard task
to the cheap tier
Each pin now records a fingerprint of the ask that wrote it, and a
continuation ignores a pin from a different ask. A new ask that cannot be
pinned also overwrites the session pin with an empty one, which covers
continuations whose history lives server side (previous_response_id) and
carry no ask to fingerprint. Pins without a fingerprint keep replaying so
rolling deploys stay consistent
Fixes#44250
* fix(complexity_router): a new user_turn ask clears the previous ask's pin
Replace the per-ask fingerprint with a simpler rule. In user_turn mode every
new ask owns the session pin: a pinnable decision writes it as before, and
one that cannot be pinned clears it, so the ask's continuations classify
instead of replaying an earlier ask's model. This also covers continuations
whose history lives server side (previous_response_id)
Housekeeping calls never clear the pin. The check reads the classified
decision before the health, modality and cache gates, which rewrite the
cause of a failed-over title call
Fixes#44250
* fix(complexity_router): hold the default model for a user_turn ask whose classifier failed
Replace the clear-on-new-ask rule with a smaller change. In user_turn mode a
classifier failure now pins the default model for that ask, so its tool
continuations replay the default model instead of the previous ask's pin.
The next new ask classifies again, so the hold ends with the ask, and
session_affinity mode keeps never pinning a failed classification
Fixes#44250
* fix(complexity_router): a plan-mode request never replays a pin without a tier
A held default-model fallback has no tier when the default model sits
outside the tier pools, so replaying it skipped plan_mode_min_tier. A
plan-mode request now classifies instead of replaying a tier-less pin,
which is what it did before the fallback was held
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): give rag ingest managed file ids and team credentials on vector store file routes
RAG ingest through a team's OpenAI deployment now registers the uploaded file as a LiteLLM managed file and returns its managed id, so GET /v1/files lists it for the caller. Re-ingesting a managed id attaches the provider file and returns the caller's managed id
Model-less vector store file calls (create, list, retrieve, content, update, delete) from a team key now use the team's OpenAI deployment that ingest uploads through, via the router, and fall back to the direct path when the key may not use one. Explicit api_key, api_base, litellm_credential_name, and forwarded provider keys still win, and blank values count as absent
* fix(proxy): route managed vector store file ids to their team deployment and keep missing-file 404s off the router
Model-less team vector store file calls now go straight to the provider with the team deployment's
credentials, so 404s on missing files no longer put the team deployment in cooldown.
A managed file id on the vector store file routes now uses the deployment named by its
llm_output_file_model_id when that deployment is proxy-wide or belongs to the caller's team, and falls
back to target_model_names otherwise.
* fix(rag): keep managed ids and team checks on every file and search path
Ingest no longer registers a raw file id when file_url is empty, a managed
file id on a proxy without the managed files hook answers 400, and vector
store search and /v1/rag/query hits show the managed id ingest returned.
Model-less vector store file calls use the team deployment only after the
key, team, and member model checks pass, fall through to the router when the
team has no OpenAI deployment, and log the deployment they used.
* fix(rag): keep provider ids on lookup failures and mint managed ids only for keys that can use them
A database error while mapping search hits to managed ids keeps OpenAI's ids instead of failing a finished search. Ingest registers a managed id only when the caller passes the team deployment's model checks, so a model-limited key keeps the raw id it can still use, and the file routes share that check.
* fix(rag): attach the managed file copy that matches the ingest credentials
* fix(vector_store_files): route a managed file id to the caller team's copy
* fix(vector_store_files): keep the upload copy for a store registered with its own credentials
* fix(vector_store_files): use a managed file's team copy only when the store holds it
* fix(vector_store_files): attach a managed file's team copy and keep named credential options out of the store check
* fix(vector_store_files): attach the team copy only to a store that routes to the team deployment
* fix(vector_store_files): keep a managed store's routing and check the upload copy through its own deployment
* fix(vector_store_files): route a managed file to its upload deployment only when the key may call it
A managed file id names its upload deployment, and with require_managed_files off that id is not checked against an ownership record. The routes now use that deployment only when the caller may call its public model, else they keep target_model_names routing as on main.
* fix(vector_store_files): log a pinned managed file call under its deployment's public model name
* fix(vector_store_files): send a pinned managed file call's deployment key only to that deployment's base
* fix(vector_store_files): keep stored credential headers intact and fail the store check on errors other than 404
* fix(vector_store_files): answer a failed store check with the provider status and copy credentials read by name
* fix(vector_store_files): send a call to the upload copy when the store check fails without a 404
A store check error other than a 404 now leaves the call on the upload copy, the way main always routed it, so the real call keeps the router's retries, the provider's status, and its failure log instead of failing on one transient error
* fix(vector_store_files): keep a credential's no-log key out of the store check
A named credential carrying a no-log key made the store check call aretrieve with no-log twice, so every managed id call through that deployment answered 500 where main answered 200
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted
* fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled
A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full.
* fix(responses): finish a turn's early token count when the client leaves
A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors
* fix(router): count one router request per Responses WebSocket turn
A socket whose finished turns each counted their own request in router
usage no longer adds one more request when it closes, so N turns count N
requests, as N HTTP calls do. A socket with no counted turn still counts 1
* fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): log keys and teams still on the deprecated empty models defaults at boot
* fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup
* fix(proxy): say which empty-models rows keep access in 2.0.0
Teamless keys listing all-team-models get no models in 2.0.0, so the
boot warning no longer says every counted row keeps its access. It
names each upgrade outcome only when its kind is in use, says the counts
include blocked and expired rows, and words each kind by what the
setting means rather than what it grants
* fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning
The warning no longer says the upgrade keeps access for empty models
lists, since no shipped code or public page backs that yet. It says
2.0.0 removes the defaults and how to keep today's access, and keeps
the advice for teamless all-team-models keys. The integration test
drops an empty-database assertion that ran before the check finished
* fix(proxy): leave rows with access groups out of the empty models boot warning
* fix(proxy): count access group rows, skip management and read-only keys in the empty models warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): list images in the Codex catalog only for models known to read them
A translated Codex entry used to offer image input whenever no deployment declared supports_vision,
so Codex let users attach images to models that may not read them. Image input now goes out only
when every deployment behind the name says it reads images.
* test(proxy): type the vision combination test parameters
* refactor(proxy): rename the vision helper and drop the extra vision docstring sentence
* feat: add customer spend breakdown and filtering by current team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: recover legacy customer team spend and tolerate lookup outages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf: resolve customer team keys once per daily activity query
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: paginate customer team activity in SQL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments
Codex builds separate catalogs for Bedrock's runtime and Mantle endpoints that turn off what Bedrock does not support yet. A model whose deployments run on Bedrock now takes its entry from those catalogs, vendored from Codex 0.166.0, instead of from Codex's OpenAI catalog.
* fix(proxy): keep Bedrock catalog entries from Codex releases that cannot decode them
codex debug models leaves out fields such as supports_parallel_tool_calls that Codex releases before 0.148.0 require, so a Bedrock entry now needs at least 0.148.0 as well as its source entry's minimum.
* fix(sso): read the SSO client secret before the refresh-token DB check so a concurrent env sync cannot split the client id from its secret
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sso): type the secret-race test and read the provider from the IdP boundary mock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(accounting): keep delivering spend when another worker's maintenance lock is orphaned
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(accounting): run worker regression tests in CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(accounting): inject fake Redis and colocate worker coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): record litellm_overhead_latency_metric for streams and count the proxy's per-chunk work in it
* fix(proxy): sum the per-chunk stream work through Logging.add_streaming_overhead so the lint gates hold
* fix(proxy): count the per-chunk work of chat completions streams too, and drop a comment that repeated the code
* fix(prometheus): keep the handoff overhead stamp when a /v1/responses stream finishes
The Responses streaming iterator calls update_response_metadata again at stream end with
include_overhead=False, which replaced the timing stamped at handoff with one that had no
overhead, so litellm_overhead_latency_metric still had no sample for streamed /v1/responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the parameters of the per-chunk overhead tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): add only the proxy's first-chunk work to a stream's overhead, not a sum over every chunk
A 1 ms hook on each of 500 chunks delays the client by 1 ms per chunk, not 500 ms, so the
overhead metric keeps the stamp plus the work on the first chunk and stops summing later ones
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drive the first-chunk overhead tests with a fake clock instead of real sleeps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): a keepalive ping is not a stream's first chunk, and test the overhead with a real ProxyLogging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust
* fix(model-cost): keep main's compat helpers pure and move network tests to integration
ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): add litellm-s3 crate, AwsCredentialsProvider and cost map fetch
* fix(model-catalog-refresh): time each read instead of the whole cost map download
The 5 s fetch timeout covered the full body, so a slow but steady link failed every attempt where httpx and boto3 would finish. Each read now gets its own 5 s, matching main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Google AI Studio answers generationConfig.candidate_count "" with a 400 before it generates anything, so the truncated 200 this case scripts is a reply Google never gives. The other malformed count and cap cases stay.
* fix(router): let a retry leave a pinned deployment after a retryable failure
The retry loop only recorded a failed deployment for non-retryable statuses,
so encrypted_content_affinity re-pinned the failing origin on every 5xx, 429
or connection-error retry. Track every deployment the request failed on and
hand that list to the callback filters, leaving selection's skip list alone.
Fixes#47607
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): carry failed deployments through stream retries too
retry_stream started each retry with no failed ids, so an affinity pin on a
stream that failed early stayed on the deployment it failed on. Seed the list
with the stream's deployment and add each retry that fails. Drop the 429
python case, whose RateLimitError mock cannot be deep-copied by the router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(router): cargo fmt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): leave out deployments that failed the request when affinity strips
When the origin had failed this request and no boundary peer existed, the
affinity check stripped the reasoning and returned every candidate, the failed
origin included. Later retries then carried no marker and re-picked the failed
deployment one time in three. Return the untried candidates instead.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-messages): apply additional_drop_params before the unsupported-param gates
* fix(anthropic-messages): apply additional_drop_params before the unsupported param checks
Share the tolerant drop list parser between the Python handler and the Rust host route, clear listed params before the Rust gate, and keep the prompt cache baseline body in step with the real request
* fix(anthropic-messages): leave additional_drop_params keys out of the Rust host body
* test(anthropic-messages): annotate the new test locals with Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): merge default metadata under the request's instead of replacing it
* fix(router): join default metadata tags after the request's tags
A router default_litellm_params.metadata.tags list replaced the request's tags, so the deployment and credential tags the router adds never reached the logged request tags. Default tags now follow the request's tags, skipping ones already present, like the proxy's key tag merge
* fix(router): keep masking a router default turns on
Merging default metadata under the request let a caller's mask_input or mask_output false switch off masking an admin set in default_litellm_params.metadata. A masking flag the default turns on now stays on, and a caller can still turn masking on. Also gives this PR's test suppressions a reason and adds a missing blank line
* fix(router): keep the router's cache namespace and drop malformed caller tags and dicts
A caller's metadata.redis_namespace no longer beats a router default, so it
cannot pick another tenant's cache partition. A caller tags value that is not a
list, or a non-dict value under a dict default, is treated as absent the way the
proxy's merge_tags treats malformed tags, so the router defaults still apply.
* test(router): check that mask_output stays on when the router default turns it on
* fix(router): keep a router default guardrail whatever guardrails the request lists
* fix(router): keep the admin's settings for a router default guardrail when the caller lists it
* test(router): check that a router default guardrail entry keeps the admin's settings
* fix(router): keep a router default private Langfuse trace private whatever the caller sends
* fix(router): keep a router default function such as langfuse_masking_function over the caller's value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Several integration tests on main still assert behavior that later changes replaced on purpose, so they fail on every scheduled run, and one of them stops the whole sdk group at import.
The batch cost case expected cached tokens to bill at the Batch input price when a price row has no Batch cache price. Since #50062 they bill at the Standard cache price, so the case now expects 0.000235 input cost and is renamed to say so.
The OTel v2 attribute test pinned langfuse_otel and langtrace as carrying no request metadata baggage. Since #50373 every OTel v2 destination carries it, so all seven callbacks share one expectation and the Langfuse baggage test that was deferred under LIT-9128 runs again.
The clean-EOF outage test trips its database relay on the text of the spend log insert. The ledger writes that insert without the "public" schema prefix, so the relay never tripped and the test timed out before it checked anything. The trigger now matches the insert on both write paths.
The usage routing tests expected cooldown, tpm and rpm keys in one MGET. Since #50170 the cooldown keys are read in their own MGET, at most once per DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS. Both the proxy and the SDK test now expect the two MGETs, and the proxy test waits out that interval after its /model/info poll, which reads the same keys.
The concurrent deployment delete test held its writer inside Router._update_deployment_indices_after_removal, which went away with the Python router core in #50178, so the file failed at import and stopped the whole sdk group. It now holds the writer where the registry reads the model list again after router-core applied the change.
Python stores a unary messages response from a background cache write task that runs after the router stamps _hidden_params, so the entry carries model_id and the router headers and a later hit returns x-litellm-cache-key. The native path stored the entry inside the native call, before the router saw the response. The async unary store is now deferred until the route encodes its public response and is handed to Python's own cache write task
When the client disconnects mid-stream, the native adapter now logs success for the chunks it delivered before teardown, so spend, SpendLogs and the key's budget and tpm reservations settle the same way they do on the Python path
The proxy records a client disconnect, with its 499 error information, in the stream generator's cleanup. The native teardown billed the stream before that cleanup, so the row had empty error information. A proxy stream torn down by cancellation now bills from the task's done callback, after the disconnect is recorded
A stream whose upstream body ends mid-frame reached proxy callers as a mapped APIConnectionError with status 500. Python's transport raises an unmapped httpx error at that point. The native body read now records a body cut and the host raises the same exception the active transport would
Python wraps a connect, DNS, TLS or reset failure in a synthesized 500 AnthropicError, which maps to InternalServerError. The native path surfaced these as APIConnectionError with the bug-report footer, which also changed cooldown decisions
The proxy bills the estimated prompt of a failed call only when the Timeout was caused by an httpx ReadTimeout. The native timeout now chains the same httpx timeout Python's client raises, so its failure row bills the prompt
Python rebuilds an upstream error response from the decoded body without content-encoding and content-length. The native failure now drops those wire headers, and a failure raised without an HTTP response states no headers at all, like Python
Python's async client wrapper stamps the effective timeout and num_retries on every failure, which the proxy reads for x-litellm-timeout and the router reads for retries. The native async paths skipped that wrapper, so error responses from Rust-served async calls had no x-litellm-timeout header
Python's async chat completions, responses, embeddings and messages entries hand the call to run_in_executor before they build the provider body, so a during_call guardrail started next to the call masks the request first. The native paths built their request without yielding, so masked text never reached the provider
The shared native_acall_hook now yields once and stamps the failure, and every async dispatch passes it as its native hook. OCR passes native_acall_hook_on_loop, which only stamps the failure, since its Python entry went through the client wrapper but built its request on the loop
Python's messages transform takes any JSON object from a 200 response and only adds stop_sequence, so a provider answering 200 with an error object reaches the client as 200 and the router counts a success. The native decoder turned it into a 500 APIConnectionError and a router failure. A JSON object that is not a message now passes through unchanged
A non-string extra_headers or deployment header value surfaced from the native handler as an APIConnectionError. It is now refused as an invalid request when the headers are collected, before the provider request is built or logged, so the caller gets a 400 BadRequestError naming the header and its type
Python logs the requested optional params, taken before the provider transform, as model_parameters. The native handler logged the transformed body, so a reasoning_effort request was logged as thinking and output_config. The request context now carries the requested params
Python updates model_call_details with the transformed request body before pre_call, so the logged messages are what was sent. The native adapter kept the caller's messages and compared params as JSON values, so an int 0 logged as 0.0
Python only records custom_llm_provider in model_call_details. The native adapter also wrote it into litellm_params, and the proxy reuses one logger across router hops, so a fallback hop's pre_call log showed the provider where Python shows null
Python only stamps cache_hit when a result came from the cache. The native adapter wrote False on every provider result
Python calls pre_call with the request body as a json user message, and a stream's spend-log cache_key is computed over that input. The native adapter logged the string Messages, so a streamed call's cache_key differed
* fix(messages): log native messages success once, after the proxy releases it
The proxy defers non-stream success logging past post-call guardrails, but the native host parked its log in its own slot and the release only ran for OCR, so a unary /v1/messages call served by Rust logged no success event, no SpendLogs row and no key, team, user or end-user spend. A deferred native success now parks in enqueue_deferred_logging with deferred_success_result, the same slots Python uses, so the proxy releases it like any other call and its guardrail step pruning and cost preparation see it too
Python runs sync success callbacks inside the deferred logging closure, so a post-call hook that blocks the response drops them. A parked native success now carries its sync callbacks and runs them on release
A refused first attempt and its fallback share one logger. Python logs the async success once, for the first attempt, so the native adapter now claims the async success log once across fallback attempts. Like Python, an attempt that loses the claim runs its sync callbacks only when logging is not deferred, and an internal call logs nothing
* test(messages): cover the deferred native success park and release
The proxy attaches its shared aiohttp session to every request and a default stream_timeout to every stream, and the messages detectors treated both as Python-only features, so no proxy request reached Rust. A shared session is only a connection-reuse hint and no native client reads it, so the common caller_client detector now fires only for an explicit client. The native messages call now takes the one timeout Python's messages handler resolves from timeout, request_timeout, stream_timeout and an httpx.Timeout, so no timeout shape keeps a messages call on Python
