* fix(proxy): redact credential headers from request logging copies
clean_headers preserves an Anthropic subscription OAuth token, and other
client-supplied provider credentials, so they can be forwarded upstream. The
same dict was also stored as proxy_server_request["headers"] and
metadata["headers"], so those credentials reached every logging callback and
the SpendLogs proxy_server_request column that the Admin UI logs page renders.
Build the observability facing copies through redact_credential_headers, and
drop the transport-only keys (provider_specific_header, headers, api_key) from
the request body snapshot since they have to keep the real values.
* fix(proxy): use the redacted header copy in the request debug log
The stdout secret filter matches Bearer and sk- shaped values, so an MCP auth
token printed by the request-header debug line survived it in cleartext.
* fix(proxy): resolve the configured MCP auth header name through the secret manager
get_secret_str also consults a configured secret manager, so a deployment that
stores the header name there now gets that header masked too. Drops the added
comments in favour of a named constant.
* perf(proxy): resolve the MCP auth header name once per process
get_secret_str issues a blocking secret-manager SDK call when one is configured,
and configured_credential_header_names runs on every proxied request.
* fix(proxy): read the MCP auth header name live, cache only the secret manager
The config reloader rewrites os.environ on an interval and after /config/update,
and MCPRequestHandler resolves the same setting per request, so caching the env
lookup left a renamed header logged in the clear until the process restarted.
Only the blocking secret-manager call stays cached.
* refactor(proxy): narrow header redaction to the reported credential set
Drops the MCP header-name resolution, its per-request config and secret-manager
lookups, and the x-mcp- prefix rule. Those cover a separate credential family
than the one this ticket reports and carried their own config-reload staleness
surface; they belong in their own change.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Team-scoped DD credentials (dd_api_key, dd_site) set via POST /team/{id}/callback were silently dropped because _request_blocked_callback_params blocks them from standard_callback_dynamic_params. The security block is correct for request-level injection, but team callback_vars are admin-configured and trusted.
Store the raw init kwargs on the Logging instance and read dd_* params from there in _process_dynamic_callback_list instead of from standard_callback_dynamic_params.
Adds an integration test that exercises the full Logging.__init__ flow with team callback_vars to prevent regression.
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
* feat(ui): expose an Auto-Router session affinity toggle
session_affinity on ComplexityRouterConfig defaults to True, and neither the
create form nor the edit modal ever emitted the key, so every auto-router built
in the UI silently pinned each session to its first turn's model for an hour
with no way to see or change that.
Adds an "Advanced: Session Affinity" switch to both surfaces, defaulted on to
match the backend field. Both paths now write the key explicitly instead of
falling through to the backend default, so a stored config states what the
router actually does. A stored config with the key absent hydrates as on, since
those routers are running with affinity enabled today; showing them as off would
report the opposite of reality and persist it on the next save.
* feat(complexity_router): default session affinity off and expose it in the UI
session_affinity defaulted to True and the Auto-Router UI never emitted the
key, so every router built there silently pinned each session to whatever model
its first turn classified into for an hour, refreshed on every hit. There was
no way to see that from the UI and no way to change it without hand-editing
config.yaml.
The default flips to False, so every turn is classified on its own merits and
lands on the cheapest adequate tier. Pinning is now opt-in.
The toggle added in the previous commit follows the field: it renders off, and
both the create tab and the edit modal keep writing the key explicitly, so a
stored config states what the router does instead of inheriting a default that
can move under it.
Behavior change for existing routers: those created before this have no
session_affinity key stored, so they pick up the new default and start
reclassifying every turn. That gives up the provider prompt cache the pin was
preserving, and a multi-turn session can now change model between turns. Set
session_affinity: true to keep the old behavior.
Key and team `router_settings.model_group_alias` was accepted, persisted and
echoed back by `/key/info`, but never applied at request time, so the request
ran on the group the caller asked for. `route_request` forwards only the
settings the Router accepts as per-request kwargs, and `model_group_alias` is
not one of them: the Router resolves aliases from its own instance attribute,
which holds the global config map and is shared across requests.
Resolve the alias in the proxy instead, alongside the existing model-alias
rewrites and ahead of the pre-call hooks, so per-model limits and guardrails
key off the group that actually serves the request. Authorize the alias target
before the rewrite; model access was checked against the requested group, so a
key whose alias points at a group it cannot call gets the usual 403 rather than
being quietly served it.
Resolves LIT-4879
Pure rename, no behavior change. create_mcp_server.tsx and its test move
to CreateMCPServer, the two importers and one stale e2e comment follow,
and the local/filename-pascal-case suppression drops now that the file
passes the rule on its own.
The rename is scoped to this one component rather than the whole
directory because three PRs are currently open against its snake_case
siblings; the rest can follow once those land.
An evicted client was left for the garbage collector, but every OpenAI/Azure
SDK client is a reference cycle, so nothing freed the client or its pooled TCP
connections until a generational sweep ran. Driving 2000 azure calls through
the official image with no forced collection, live clients and open sockets
climbed from 202 to 1361 while the cache stayed at its 200-entry bound, and RSS
grew 279 MB to 456 MB against a TLS upstream.
Closing on eviction is what caused the earlier 'Cannot send a request, as the
client has been closed' regression, so an evicted client litellm created is now
closed only once a grace window has passed, by which point any request that was
already holding it has finished. A client the caller supplied is never closed,
since litellm does not own its lifecycle.
Resolves LIT-4883
* feat(teams): apply default organization to new teams from default team settings
Adds organization_id to DefaultTeamSSOParams so proxy admins can pick a
default organization in Default Team Settings. new_team applies it before
org validation whenever a team is created without an explicit
organization_id, so API, Admin UI, SCIM, SSO, and team upsert creations
all inherit it and go through the same existence and org-limit checks.
Explicit organization selections win and existing teams are untouched.
The default is validated at save time (PATCH /update/default_team_settings
returns 400 for an unknown org) and at create time, where a missing org now
surfaces as a clean 400 instead of a 500 by routing OrganizationNotFoundError
into the previously dead org_table None guard.
The Admin UI Default Team Settings tab gets a Default Organization row
backed by the shared OrganizationDropdown.
* fix(teams): validate org limits against final team state including defaults
Applies default_team_params and the legacy max_budget fallback before the
organization validation block, so _check_org_team_limits sees the values the
team will actually be persisted with. Also loads the org's budget table in
the lookup; without include_budget_table every budget comparison in
_check_org_team_limits was skipped because litellm_budget_table was None.
* test(proxy_behavior): pin org team limits as enforced on /team/new
The dead-code pins existed to turn red when include_budget_table went
live; that happened, so the scenarios now assert the 400 rejections plus
within-cap acceptance, and the unknown-org pin asserts the handler's 400
instead of the surfaced 500.
* fix(proxy): backfill null user_email on existing users during JWT auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): guard mapped-key email backfill and make null update atomic
Resolve Greptile review on the JWT user_email backfill:
- only backfill when the mapped virtual-key owner is the JWT principal, so a
mismatched admin-created mapping cannot write one user's email onto another
- make the best-effort mapped-key enrichment non-fatal so a database outage on
a cached-key request no longer fails otherwise-valid authentication
- persist the backfill with an atomic null-guarded update_many so concurrent
writers cannot overwrite an already-populated email
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep cache coherent when a concurrent backfill wins the null-email update
* fix(proxy): cache DB-persisted email after JWT backfill, not the proposed value
Resolve the Greptile finding that a successful null-guarded backfill could
cache this request's proposed email even if a concurrent ordinary user update
wrote a different email first. The helper now always re-reads the row after the
atomic update and refreshes the cache from the value the database holds, so
cache-hit auth and attribution stay consistent with the persisted record.
Annotate the Prisma and model_copy dict literals to keep the LIT002 budget within its ceiling.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
disable_team_logging cleared only metadata["callback_settings"], but callbacks
registered through POST /team/{team_id}/callback and the Admin UI live in
metadata["logging"], and request-time resolution stops at that slot without
ever reading callback_settings. The endpoint reported success while the team
kept sending request and response data to its third-party destination.
Empty the logging slot alongside the existing callback_settings reset, and
refresh the cached team object so the change applies to keys that are already
in flight rather than at the next cache expiry. The same refresh is added to
add_team_callbacks, which has the symmetric problem of a newly registered
callback staying dormant until the entry expires.
Resolves LIT-5101
Bedrock managed-batch file upload read `messages` unconditionally, so a
JSONL record shaped for /v1/completions (`prompt`) or /v1/responses
(`input`) reached the per-provider transform with an empty message list.
Anthropic and Nova rejected it at POST /v1/files, and the passthrough
providers shipped an empty conversation to AWS.
Classify each record by its OpenAI batch `url`, then normalize the
non-embedding shapes to chat completions before the Bedrock transforms:
`prompt` wraps into user messages the way litellm.text_completion does in
real time, and `input` goes through the existing Responses-to-Chat bridge.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LiteLLM_ManagedObjectTable only stores created_by (user_id) and team_id,
never the raw API key hash. A batch created with the master key or a
team-less key has both null, so CheckBatchCost's synthetic logging_obj
for the completed batch carried no attributable key/user/team/end-user.
_should_track_cost_callback silently skipped the DB write in that case
(by design, to avoid tracking truly anonymous requests), with no error
or warning: batch_processed still became true, but no LiteLLM_SpendLogs
row was ever written despite real, already-incurred provider cost.
Extend the same allowance already made for unauthenticated pass-through
requests to aretrieve_batch's cost event, and pass job.team_id through
so a batch's team gets real attribution when one exists.
The strict-priority e2e (added with the zero-increment limiter fix) can
never pass on stage: the proxy there does not run the
dynamic_rate_limiter_v3 callbacks + priority_reservation settings the
module requires, confirmed by zero limiter log lines across every
gateway and backend pod during the 2026-08-02 run. Config lives in the
infra repo; LIT-5118 tracks adding it.
The throughput SLO test failed the same run with 65.9% of requests dying
at the ELB as 502/503 before reaching a pod. The per-replica SLO rework
fixed the RPS-floor assertion but cannot help when stage idles at one
warm gateway replica; LIT-5119 tracks pre-scaling the fleet for the load
phase.
Both skips name their ticket, and the coverage registry returns the two
cells to the gap list while they are in place.
A single read of key_info.spend races the batched spend writer: deltas
earned before a reset flush to the DB up to ~60s later
(proxy_batch_write_at) and land on the row after the reset zeroed it.
The stage runs on Jul 30 and Aug 2 failed
test_key_budget_reset_at_advances_after_window exactly this way, with
spend back at the driven total while budget_reset_at had advanced and
calls flowed again.
Replace the single reads in rung 3 (spend zeroed after reset) and rung 4
(roomy window keeps spend) with _poll_key_spend, which re-reads to a 90s
deadline covering one full flush-plus-reset cycle. A reset that never
zeroes the row keeps spend pinned and still times out, so the regression
guard keeps its teeth.
Fast mode is priced with a provider-specific multiplier applied off usage.speed, but only chat completions kept that field. The Messages route rebuilt usage with empty optional params, stream reassembly dropped speed and inference_geo, and the pass-through handler never read speed off the request body, so fast-mode spend was logged at the standard rate.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cursor appends -thinking-<level> and -fast to custom model names when the
user picks a thinking level or fast mode, so a model configured as
claude-opus-5 arrives as claude-opus-5-thinking-xhigh-fast and fails
routing with no healthy deployments. When the raw name is not servable by
the router but the suffix-stripped base name is, rewrite the body to the
base model and carry the thinking level into reasoning_effort (chat
bodies) or reasoning.effort (Responses bodies), never clobbering an
effort the client already sent. Explicitly configured aliases keep
winning because the raw-name servability check runs first.
* fix(team-callbacks): report API-registered callbacks from GET /team/{team_id}/callback
POST /team/{team_id}/callback writes metadata["logging"] while the GET read
metadata["callback_settings"], so every team configured through the API or the
Admin UI got back an empty list. c620d76fe4 migrated the writer to the new key
and left this reader on the old one.
Resolve the read the same way request-time resolution does in
_get_dynamic_logging_metadata: a logging slot that is present wins outright and
callback_settings stays as the deprecated fallback, so the endpoint reports what
a request would really do rather than the union of both shapes. An empty logging
list therefore reports no callbacks, matching a request that fires none.
Decrypt callback_vars for the response and mask the credential keys. Ciphertext
would be unusable to the caller, and a value encrypted under a key that is no
longer classified as sensitive would otherwise come back as a raw blob.
Resolves LIT-5093
* Update litellm/proxy/management_endpoints/team_callback_endpoints.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team-callbacks): mask callback vars that fail to decrypt
decrypt_callback_vars passes a value through untouched when it cannot be
decrypted, which happens to existing rows after a salt-key rotation. Under a
key that is not classified as sensitive that blob reached the caller as opaque
ciphertext it could not use or tell apart from a real value, so mask anything
still carrying the encrypted prefix.
Raised by Greptile on the first commit.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A keyless internal user signing in to the Admin UI was redirected off the
post-login landing to /ui/connect, which renders nothing but the MCP apps panel,
so a plain gateway sign-in ended on an MCP OAuth surface the user never asked
for. The landing now renders the keys dashboard for every role. The key lookup
that existed only to make that routing decision goes with it, along with the
useKeys enabled flag it was the sole caller of and the role-hydration hold that
guarded its one-frame dashboard flash
The gateway DCR consent flow moves the other way. Its /authorize handed the
browser to /ui/chat/integrations, whose layout hard-blocks when enable_chat_ui
is off, which is the default, and client-side redirects to /ui/ without the
query string; that destroys the connect_flow handle and strands the MCP client
until the 600s flow cookie expires. It now lands on /ui/connect, which reads
connect_flow and connect_client, mounts the consent banner and puts the apps
panel in connect mode. /ui/chat/integrations keeps its connect-mode handling
this release so flows sealed before the deploy still finish
Resolves LIT-5104
Resolves LIT-4911
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.
B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.
ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
Two changes to the classifier's system role, both narrowing it rather than adding to it
classifier_tier_rubric let an operator replace the tier definitions. It shipped in
#35471 alongside the assistant-turn context window, but the two answer different halves
of the same report and only the context window was asked for. The override carried a
composed prompt, an overridable and a non-overridable half, a blank-is-unset rule, a
length-warning validator and a pair of dashboard controls. All of it goes
The rubric then closes on one of two lines, chosen by classifier_context_window_size.
At 0 no conversation is quoted, so the line is the original one, byte for byte: a
deployment that sends no context is told to classify the current message and nothing
else, which is what it could see all along. Above 0 the turns are quoted, and the
original line told the model to disregard them, which is how a request whose difficulty
was established in an earlier turn came back SIMPLE on the word "yes". There the line
instead says to classify the current message using the quoted turns as context, and to
rate what a short reply approves rather than the reply
The choice keys on the window and not on classifier_context_include_assistant_turns.
Whether the quoted turns are the user's alone or include the assistant's replies does
not change what the model needs told, and whose turn is whose is already on the turns.
Keying it on the assistant toggle would put the default deployment back on the original
line, which is the configuration the report was raised against
Folds in #35508, which built the window-dependent framing on top of the override this
removes; that PR is closed in favour of this one
`import litellm` reaches litellm/integrations/otel/model/config.py via
litellm_core_utils/litellm_logging.py, so pydantic-settings is needed at import
time. It was declared only in the `proxy` extra, which left a plain
`pip install litellm` unimportable on every platform.
Adds tests/base_sdk_tests/check_base_sdk_install.py and a base_sdk_install
CircleCI job that builds the wheel, installs it into a clean venv with no extras,
and smoke-checks the import, a mock completion, a mock embedding, the bundled
pricing metadata and the token counter. The check is stdlib-only on purpose;
installing pytest into that venv would add packaging, pluggy and iniconfig and
could mask the class of undeclared dependency it exists to catch.
Previously the Windows job was the only one installing without extras, so this
class of break was caught by accident rather than by design.
Gating the mock testing request params behind
general_settings.dangerously_allow_mock_testing_request_params (#35423) turned
every fallback, retry and timeout drill in tests/test_fallbacks.py into a 400:
the build_and_test job mounts proxy_server_config.yaml, which never opted in.
Opt that config in. It is the config the CI proxy runs with, and the suite it
serves exists to drive synthetic failures.
Add a unit test that ties the two together: it scans the top-level tests/test_*.py
files build_and_test globs for gated param names and fails if the config they run
against has not opted in, so the next change to either side is caught in a fast
lint-tier job rather than a Docker E2E.
Adding a team member by a user_id with no user row is now proxy-admin-only,
so the /team/member_add authz matrix, which targeted a never-seeded user_id,
started 403ing every non-proxy-admin caller. Seed the member as a real user
row so the matrix reads _validate_team_member_add_permissions alone; leaving
it unseeded and relaxing the expectations to 403 would have left all 18 rows
green with that gate deleted outright.
Cover the new gate at the HTTP boundary, where only the helper was pinned
before: a team admin and an org admin both clear the permission check on the
same team and are still refused an unprovisioned user_id, with no user row
left behind. Pin the escape hatch that refusal names too, so closing the
email-invite path for non-proxy-admins cannot pass silently.
Promote the user seeder the member-info pins had kept private to conftest,
and reclaim invited users by their scratch-prefixed email, since an invite
allocates the user_id server-side.
_delete_deployment stopped returning a count of evictions in #35400 and now returns
the frozenset of ids the db and config still want, so a caller judging its own reload
can tell a deliberate eviction from a deployment that went missing. These two tests in
tests/local_testing were left comparing that frozenset against an int and have been
failing since; the directory is only referenced by .circleci/config.yml, which no
longer reports checks on PRs, so nothing caught them.
The eviction behavior itself is unchanged, so the fix is on the assertions: compare
against the expected id set, and pin the router's surviving ids so a mutation that
evicts the wrong deployment is caught rather than passing a bare length check.
The componentized images exec uvicorn directly, so ddtrace-run never wraps the
interpreter. USE_DDTRACE is not inert there; the proxy lifespan still runs
patch_all and litellm's own manual spans still emit. What never gets installed
is ddtrace's ASGI TraceMiddleware: starlette builds its middleware stack lazily
on the first __call__, which is the lifespan scope, so patching from inside the
lifespan body is already too late and no root request span is ever created.
Route both entrypoints through a shared docker/component_entrypoint.sh that
mirrors the monolith's prod_entrypoint.sh contract, including the
DD_TRACE_OPENAI_ENABLED=False export that keeps ddtrace's openai integration
from double-reporting calls litellm instruments itself.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
The migrations image ran `prisma migrate deploy` against a bake anchored in
$HOME with no node in the runtime stage, so prisma-client-py fell through to
nodeenv and tried to download a Node runtime on first start. In an
egress-restricted cluster that fails outright, and under an arbitrary uid the
uid-specific cache path is unreadable, so the job never applies a migration.
Move the bake to /opt/prisma with world-readable modes, install node in the
runtime stage, and pin PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH /
PRISMA_OFFLINE_MODE so the migration entrypoint runs the cached CLI directly.
This is the same treatment the root, non_root and database images already
carry.
Resolves LIT-4727
The floor was an absolute fleet number, so it asserted replicas x per-replica rate
and went red on how many gateway pods happened to be warm rather than on the request
path. The test now measures one replica first, with a short serial pass that only ever
occupies a single pod, and requires the concurrent phase to reach at least that rate.
A serial latency budget carries the request-path assertion the floor used to imply,
and both hold at one replica or seven.
Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load
model is a mock_response deployment with no upstream, a single-worker replica serves
it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at
6s each by Little's law.
The runner also kept locust's --json summary and threw away everything else, so a run
where 93% of requests failed said nothing about what they got. It now passes --csv,
reads the failure breakdown back, and reports locust's own generator-saturation
warnings, both folded into the assertion messages.
Resolves LIT-5054
The LLM classifier's context window carried user turns only, so a conversation
whose difficulty was stated by the model rather than by the user was classified
without it. Asked to find events, the assistant answers "here is the plan, it is
complex, should I execute?", the user answers "yes", and the router rates the
word "yes" and picks the cheapest tier
Two independent causes, so two changes that are each provable on their own
classifier_context_include_assistant_turns adds assistant turns to the window.
It is off by default because turning it on shifts tier decisions, and therefore
spend, for an already-deployed router, and because assistant text is net-new
egress to the classifier deployment. With it on, classifier_context_window_size
counts the last N turns across both roles, which is what makes the assistant's
own statement of difficulty land in the window
Assistant text reaches the classifier payload and nothing else. The window is
read only by _build_classifier_user_payload, while keyword_tier_rules, escalation
matching, the heuristic scorer and the semantic embedding all read the human ask
through _iter_human_asks_newest_first. Those are substring and vector matchers,
so an assistant echoing an escalation keyword back to a user would choose the
model, and the spend, with nobody having asked. Rather than widen the shared
iterator, _iter_context_turns_newest_first is separate and feeds the window
alone, which makes the boundary structural instead of a rule to remember
The rubric ended "Classify only the current message", and the classifier applied
it literally: a request whose difficulty was established earlier came back SIMPLE
because the message being rated was the word "yes". A context window the rubric
then tells the model to disregard buys nothing, so the wording now asks it to
rate the work the current message approves, judged in the conversation it
continues, while still forbidding it to rate a quoted section as if that section
were the request
classifier_tier_rubric lets an operator replace the tier definitions. The
trust-boundary paragraph is appended and cannot be replaced: it defends the
operator against their own callers, so an operator writing tiers without that
threat in mind would otherwise hand every keyholder the top tier by omission.
Blank reads as unset so an empty form field falls back rather than sending a
rubric with no tiers in it
Turns are labelled by role only when assistant turns can appear, so the prompt of
every deployment that never asked for this is unchanged byte for byte
The explicit AssumeRole branch of BaseAWSLLM.get_credentials returned without
touching the process-wide IAM cache, so every model request issued a fresh
sts:AssumeRole, and on ECS/EC2 an uncached sts:GetCallerIdentity ahead of it.
Route the whole role branch through _get_or_set_cached_credentials with the TTL
_auth_with_aws_role already computed and discarded. The cache key is the same
aws_* argument snapshot the other flows use, taken before the session-name
default is filled in, so each aws_session_name keeps its own STS session and no
attributed identity can be served another's credentials.
Credential fetches now single-flight behind striped locks. Without that, a burst
of concurrent misses on one key each issued their own STS call, which is the
same thundering herd the cache exists to prevent, moved to the miss window.
A daily-spend batch upsert that outlives prisma-client-py's 30s HTTP read
timeout keeps running server side after the client gives up, holding its
row locks for as long as the database takes. Every later flush cycle
queues behind those locks, which is how one slow batch cascaded into
exhausted database sessions.
The query engine's own transaction timeout cannot end that wait: it
cannot interrupt a statement that is already executing. Measured against
real Postgres, a batch wrapped in db.tx(timeout=60s) still held its locks
for the full 90s the statement ran. Only a Postgres-side statement_timeout
bounded it.
database_statement_timeout and database_lock_timeout (seconds) are now
first-class general_settings keys, emitted as libpq
options=-c statement_timeout=<ms> on DATABASE_URL. They are opt-in, so an
unset config keeps today's behavior, and they are never applied to
DIRECT_URL, which serves migrations that legitimately run long.
Resolves LIT-4718
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
The Prisma query engine is a separate Rust process whose resident memory is
a high-water mark: it grows with the payload of the largest single statement
it executes and glibc never returns that memory to the OS, so a pod's memory
floor ratchets up to its worst-ever write and stays there for the life of
the worker. Memory-based autoscaling then reads a number that reflects the
largest write the pod has ever done rather than what it is doing now.
The spend-log flush handed Prisma a fixed 1000 rows per create_many. With
store_prompts_in_spend_logs enabled a single row carries the full prompt and
response, so one statement can be tens of megabytes and permanently costs
hundreds of megabytes of RSS. Row counts cannot express that budget: the
same 1000 rows range from well under a megabyte to tens of megabytes.
Split each flush into statements bounded by encoded payload size
(SPEND_LOG_WRITE_BATCH_MAX_BYTES, default 2MB) on top of the existing
1000-row cap. What is measured is the encoded statement, so the budget
counts what actually goes on the wire: the JSON escaping of quotes and
newlines, multibyte characters at their encoded width, the field names and
separators a 25-column row carries, and the brackets and row separators the
rows carry as one collection. Deployments that do not store prompts keep one
statement per 1000 rows and are unaffected; prompt-carrying flushes get
several small statements instead of one huge one. A row larger than the
budget is still written on its own rather than dropped, and a row the
serializer refuses counts as zero rather than raising out of the flush and
dropping every row queued behind it.
Splitting a flush must not multiply what a poison-row flood costs, so the
poison-isolation allowance is threaded through every statement of a 1000-row
group instead of being handed out fresh per statement. That is only safe
because the allowance now counts failed inserts rather than every insert:
the one insert a statement needs when nothing is poisoned is not charged, so
a healthy flush never runs the allowance down however many statements it
splits into, and a statement reached after the allowance is spent is still
attempted so clean rows behind a flood still persist. Failed inserts for a
group are bounded by the allowance plus one baseline insert per statement,
which restores the constant-per-group ceiling the single-statement path had.
Resolves LIT-4765
A failing deployment stamps its own litellm_params.num_retries onto the raised
exception, and async_function_with_retries adopted that value unconditionally. So a
model_list num_retries outranked both the x-litellm-num-retries header and the request
body, inverting the documented precedence to model_list > header > body >
litellm_settings.
The router could not tell a request-level value from its own default because the entry
points filled num_retries in with self.num_retries whenever the caller omitted it,
collapsing "the request asked for N" and "nobody asked". Drop that pre-fill from
_update_kwargs_before_fallbacks and from the six entry points that also did it a line
above their own call to it (image generation sync and async, adapter completion, file
create, batch create, batch cancel), all of which reach async_function_with_retries,
where the router/global default is already resolved. Leaving them would have made the
request value never None on those routes and permanently suppressed a deployment
num_retries there.
The sync text_completion pre-fill stays. That path resolves a deployment and calls
litellm.text_completion directly, never entering the retry loop, so no request-versus-
deployment ranking happens there and there is nothing to fix; removing the line would
only change which value is forwarded to litellm.text_completion, a behaviour change this
bug does not call for.
async_function_with_retries then adopts the deployment's value only when the request
carried none. Precedence is now header > body > model_list > litellm_settings, with the
deployment value still beating litellm_settings when the request is silent, on every
entry point that retries.
Resolves LIT-4772
The public A2A guide tells users to declare agents under a top-level
`agents:` key, but the proxy only ever read `agent_list:`, so the
documented config was silently ignored and GET /v1/agents returned an
empty list. Accept `agents` as the documented spelling and keep
`agent_list` working for anyone who found it by reading the source.
Selection is by key presence, so an explicitly empty `agents: []` is not
overridden by leftover legacy entries.
Config-defined agents were also dropped on any database-backed gateway:
the periodic reload rebuilt the registry from the DB rows plus a module
global that was declared and never assigned. The registry now remembers
the agents it loaded from config.yaml and replays them on every rebuild.
A database row wins a name collision, mirroring how config-declared MCP
servers are unioned under the database registry, so name lookups and
deregistration keep addressing exactly one agent.
Resolves LIT-4978
OpenAI cut Terra 20% and Luna 80% on 2026-07-30; openai and bedrock_mantle
entries already match. Azure global and us/eu data-zone terra/luna rows still
used the pre-cut rates, so spend tracking over-billed those Azure deployments.
Sol is unchanged. Cache-read, priority, and long-context fields scale with the
same multipliers already used for azure gpt-5.6.
