ollama_chat streams tool calls in a chunk with done false and then sends a
separate done chunk whose message is empty and whose done_reason is stop. The
override that turns the finish reason into tool_calls only looked at the done
chunk's own tool_calls field, which is None by then, so the stream ended in
stop.
Clients that gate tool execution on finish_reason == tool_calls therefore
threw away the tool_calls delta they had just accumulated and treated the turn
as ordinary text. Non-streaming was fine; it reads message.tool_calls off the
full response.
Track whether any chunk in the stream carried tool calls and use that on the
done chunk. Streams that never produce a tool call keep whatever done_reason
Ollama sent, so stop and length are unchanged.
* feat(otel): stamp service tier attributes on inference spans
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): bound requested service tier to known values
The requested tier is caller-controlled and reaches the span verbatim, so an
arbitrary string lands on every litellm_request span on success and on failure.
A 100k character value was stamped uncapped; safe_set_attribute does not
truncate and no span limits are configured.
Apply KNOWN_REQUEST_SERVICE_TIERS in get_requested_service_tier so both the
span attribute and the Prometheus label bound the value the same way. The
served tier stays unrestricted since it comes from the provider, so a tier a
provider adds later is still reported.
Prometheus label behavior is unchanged.
* fix: derive known service tiers from the ServiceTier enum
The allowlist omitted "fast", which litellm models as a real tier and prices
through the priority cost key, so a request naming it resolved to no tier on
the span and no Prometheus label.
Deriving the set from ServiceTier keeps the two in sync, so a tier added there
for cost calculation cannot go missing here.
Behavior change: a request with service_tier "fast" now carries the tier on the
span and on the Prometheus service_tier label, where it previously resolved to
none. Every other value resolves as before.
* refactor: build the known service tiers without a mutable intermediate
The set comprehension and set literal tripped LIT002, which bounds mutable
collections. Concatenating tuples keeps the derivation from ServiceTier while
every intermediate stays immutable; the resulting frozenset is unchanged.
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
* fix(proxy): retry model cost map fetch with Retry-After-aware backoff and stop downgrading to the packaged backup on reload failure
A 429 or transient network error during a manual or scheduled model cost map
reload used to silently replace litellm.model_cost with the stale backup JSON
bundled in the installed wheel, stamp the reload as successful, and clear the
force_reload flag, so a fleet could serve months-old pricing until the next
interval. Runtime reloads now go through refetch_model_cost_map, which retries
429/5xx/transport errors up to 3 times honoring Retry-After (capped at 30s,
exponential backoff with jitter otherwise) and returns a failure value instead
of the backup when the fetch or integrity validation fails. On failure the pod
keeps its currently loaded map, the periodic job leaves last_run and
force_reload untouched so it retries on the next config poll, and the manual
endpoint returns 502 with the reason instead of reporting a fake success.
Startup behavior is unchanged: boot still falls back to the packaged backup
since there is no previously loaded map to keep.
* fix(proxy): use shared async httpx client for cost map reload and make retry tests CI-env-proof
The reload fetch now goes through get_async_httpx_client with a dedicated
httpxSpecialProvider.ModelCostMap pool instead of constructing a raw
httpx.AsyncClient, so it inherits deployment-level TLS and transport settings
and passes the ensure_async_clients gate. Tests inject a MockTransport-backed
client through the same seam. An autouse fixture clears
LITELLM_LOCAL_MODEL_COST_MAP, which CI exports and which short-circuited the
retry tests; the two TestPriceDataReloadAPI tests and the config sync pubsub
reload test that still patched get_model_cost_map now patch
refetch_model_cost_map instead.
The auto-router savings driver recomputes what the served request cost, but that
request is not a counterfactual: it ran, and the cost calculator already billed it and
wrote the number down. Recomputing means restating every pricing dimension the biller
applied, and the two this missed were enough to halve it. A request billed at a
priority tier is recomputed at standard rates, and a regional host's uplift is dropped
entirely, so the driver writes a savings figure into the same rollup row as the `spend`
it disagrees with. On `gpt-5.4-mini` at priority the row is billed 0.024 and the driver
prices the same usage at 0.012.
Neither omission cancels between the two arms, because both are per-model. The uplift
is a multiplier read off each model's own entry, so 1.1*A - 1.1*B is 1.1*(A-B) and a
model without one does not move at all. Tier coverage is sparser and asymmetric:
`gpt-5.6` has priority rates and `gpt-5.4-nano` has none.
`cost_breakdown` already carries the answer and already reaches the call site. The cost
calculator records it, it rides the standard logging payload into the spend log's
metadata, and OTEL, the log drawer and the response headers all read it rather than
re-deriving; this driver was the only downstream consumer in the tree still pricing a
completed request from its tokens. `input_cost` and `output_cost` sum to exactly what
the pricer returns, so the served arm reads them. Tool spend, discount and margin stay
out, since the counterfactual cannot be priced with them and charging them to one arm
alone would read as the router losing money on every tool call.
The baseline never ran, so it is still priced through the cost engine, now on the basis
the biller used. `CostBreakdown` carries that basis because it cannot be recovered
afterwards: the tier the biller used comes from `optional_params`, which no log record
keeps, and the served tier that does survive on the usage object is a different fact
with the opposite precedence. Rows written before this shipped carry no basis and price
at standard rates, exactly as they do today; there is no backfill.
Two smaller things in the same path. The router is passed as a provider rather than a
router, so a spend write that was never auto-routed no longer fetches and discards one,
and the complexity router resolves its messages once per hook instead of once per
consumer.
* feat(spend): add net auto-router savings to the cost-optimization dashboard
The dashboard credited compression and prompt caching but said nothing about the
optimization that picks the model, so the driver with the largest lever on a bill
was the one an operator could not see.
Savings are the counterfactual: without a router a deployment runs one model, and
it has to be one that can carry the hardest request, so the baseline is the
priciest model in the router's hardest configured tier. A cheap tier is a choice
the router made, not a ceiling it was bounded by. `auto_router_savings_baseline_model`
overrides it for operators who would genuinely have run something else. Both are
provider-qualified before pricing, because a bare name can resolve to a different
vendor's rates or to nothing at all, and a deployment is priced by its `base_model`
where it has one, which is how Azure deployments are priced everywhere else.
Both arms price the request's real usage through `generic_cost_per_token` rather
than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers
and regional uplifts stay consistent with what was actually billed. `prompt_tokens`
already includes the cache buckets, so charging them again at the input rate would
price the same tokens twice.
Cache state is what makes this hard. The baseline serves every turn, so whether it
had the prompt cached is whether the conversation was already underway. On a
continuing conversation it wrote the prompt earlier and would only read it now, so
this request's write is what switching cost and counts against the saving. On a
first turn nothing was cached for any model, the baseline would have written the
same prompt, and both arms carry the write at their own rates. Charging the write
to both cases understates a first turn to a few percent of its value, and because
the write premium is fixed by prompt size while the saving grows with completion
length, it can render a profitable route as a loss.
That shape is read off the conversation rather than remembered: a second human ask
means an earlier turn was served. No cache, no session id, and no dependence on a
caller sending a session header. It cannot see a switch on a turn the router did
not classify, and it reads a few-shot prompt's synthetic turns as prior
conversation; both err toward charging the write, which under-claims.
The baseline and the shape ride on the existing `routing_decision` record, which is
already carried from the router to the spend log, already classified for redaction,
and already written-or-cleared per attempt. A fallback that re-enters the hook
therefore cannot leave either fact behind to be attributed to a deployment that
never routed, and no new metadata key crosses the trust boundary.
The result is signed. Whether a switch pays off is a race between the rate gap and
the cache-write cost, and a narrow gap loses; flooring at zero would hide exactly
the routing behaviour an operator needs to see. The donut plots only drivers that
saved, while the card and range total keep the sign.
Savings accrue into a new `autorouter_savings_spend` column on the six daily rollup
tables, declared `NotRequired` because rows queued by a pod on the previous release
carry no such key. It is summed by the rollup merge the cross-pod Redis drain also
runs, and carried through the aggregation query, the per-row accumulation and the
response model, so the dashboard reads a value the API actually sends. Tests
enumerate the drivers from the response model itself and assert each is summed,
accumulated, carried and totalled, so one added later cannot be half-wired.
* fix(spend): let the baseline pay for a continuing turn's own growth
`_baseline_usage` moved every cache-creation token into the baseline's read bucket
whenever the conversation was underway. That is right for a switch, where the
baseline never left the model it was on and really would only read, but wrong for a
turn that stayed put: the prompt grew, and the tokens written are that growth. They
are new to every model, so the baseline would have paid to write them too. Forgiving
it that write made the counterfactual cheaper than it was and shrank the reported
saving on ordinary steady-state traffic, by about 2% per turn.
The selected arm was never involved; it has always been priced on the real usage.
The error sat entirely on the baseline.
The condition is that the request read more than it wrote, not that it read anything.
A switch onto a model already holding a small prefix of this prompt still writes most
of it, and that write is the switch's own cost; keying off a nonzero read would have
handed such a request the full rate gap, turning +$0.0056 into +$0.1177. Comparing
the two buckets separates a warm continuation, which reads far more than it writes,
from a cold arrival, which does the reverse, and it leaves the existing invariant
intact: a request reading 0 and one reading 1 both still land in the same place.
* fix(spend): price each arm under the key litellm billed it, and see agent turns
Two ways the savings number read the wrong thing, both from identifying a model by
its name when the name is not what it costs.
The counterfactual was ranked and priced on the public rate for the model a
deployment names. A deployment may not be charged that rate: the router registers
its configured prices under the deployment's own id and deliberately keeps them off
the shared model-name key so deployments sharing a backend model do not pollute each
other. So a hardest-tier deployment configured above its public rate lost the
ranking to a cheaper candidate, and once chosen was priced at a rate nobody pays.
Which key prices a deployment is now `_select_model_name_for_cost_calc`'s decision,
the resolver the real request is billed through, rather than a second rule here that
would have to re-learn that per-second and tiered overrides count, that a partial
override still counts, and that a deployment configured at zero is priced at zero
rather than treated as unpriced.
The arm being subtracted had the same fault and a sharper edge. It priced the spend
log's `model`, which on Azure is the deployment name, absent from the cost map, so
the whole driver silently read zero for that traffic. It no longer re-derives
anything: `model_map_information.model_map_key` is what litellm actually billed the
request under, recorded at request time by that same resolver with `base_model` and
custom pricing already applied.
Separately, the conversation-shape discriminator counted human asks, and an agent
loop can run twenty turns on one of them. Its tool traffic rides `tool_result`
blocks on user turns that flatten to empty text, and `tool` roles that are never
read, so a long agentic conversation looked like its own first turn and was handed
the arithmetic that leaves the cache write on both arms. That is the one direction
this must never fail in, because it inflates. An assistant turn is the direct
evidence that something answered earlier, and it is blind to how the tool plumbing
is spelled on either surface.
* fix(spend): give the cost-key resolver both inputs the selected arm needs
The served model was resolved through one input at a time, and each choice broke the
half the other fixed.
`model_map_key` is the served model already resolved through `base_model`, which is
the only way an Azure deployment name reaches the cost map at all; without it the
selected arm priced a name absent from the map, returned nothing, and the whole
driver silently read zero for that traffic. But it is built without
`router_model_id`, so it never carries a deployment's own price overrides, and a
custom-priced deployment was compared at its public rate while the baseline used the
real override. On a deployment configured well above its public rate that inverted
the answer outright: a route that lost $21.88 reported saving $0.10.
`_select_model_name_for_cost_calc` takes both, so it gets both. Which key prices a
deployment stays its decision rather than a rule restated here.
* fix(spend): same model is only the same cost when it is the same deployment
The short-circuit compared resolved model identity, so two deployments of one model
collapsed to "no switch" and reported zero. They are not the same cost: a deployment
can carry a negotiated rate, and routing from the dear one to the list-price one is a
real saving the dashboard reported as $0.00 against a true $21.93.
Both arms now carry the key litellm prices them under, so the comparison is between
deployments rather than between names.
* refactor(spend): price from resolved rates, not from a name we keep re-resolving
Four review rounds landed on one mechanism: which identifier prices a deployment.
base_model, then the deployment id, then cache-only overrides. Each round added a
clause to a resolution rule that should not exist, and a wrong primitive fails once
per input shape, so each shape arrived as its own finding.
`Router.get_deployment_model_info` already owns this. It merges a deployment's
configured prices over the built-in map, folds in `base_model` defaults for
deployments whose name is not a model, and falls back to the model name when nothing
is overridden. Every shape hand-rolled here (cache-only, partial, per-second, Azure)
was that function re-implemented badly.
`generic_cost_per_token` now accepts already-resolved rates instead of demanding a
name it looks up itself, which is what forced the name-bending in the first place.
Both arms resolve through the owner and pass what they got: the counterfactual by the
deployment the router would have used, the served request by the deployment that
served it. The invented cost-key resolver is gone, and `Baseline` carries a
deployment id rather than a key we chose on litellm's behalf.
Net 64 insertions against 79 deletions.
* test(spend): follow _most_expensive onto the router that prices its candidates
Ranking moved through `Router.get_deployment_model_info`, since what a deployment
costs is the router's answer to give; these four cases were still calling the old
free-function signature.
* fix(spend): rank baseline candidates by what a request costs, not by two rates
"Most expensive" was decided by comparing output rate then input rate. That is a
property of a rate, not of a request: a deployment dearer per output token can be
cheaper per cached token, so the comparison ordered cache-heavy traffic backwards and
recorded the wrong counterfactual.
Candidates are now costed on one reference request through the same engine the
savings themselves use, which leaves cache read and write rates, tiered tables and
every other billing dimension to that engine rather than to another rule restated
here. The reference request is cache-heavy because auto-routed traffic is.
* fix(spend): pick the baseline against the request that ran, not a stand-in for one
Ranking happened in the pre-routing hook, where the request has not executed yet, so
candidates were costed against a hard-coded reference workload: 20k prompt, 19k of it
cached, 1k out. Which candidate is dearest depends on that mix, so a pooled hardest
tier holding a deployment with non-proportional configured rates could be ranked for
a request nothing like the one served.
The mix is known on the spend path, so the ranking belongs there. The routing
decision now carries the tier's candidates rather than a winner already chosen, and
the baseline is resolved against the usage that actually happened. The reference
workload is gone; nothing here assumes a traffic shape any more.
The router is passed in rather than imported from `proxy_server` inside the
computation, so the savings stay a pure function of their arguments and the caller
owns where the router comes from. That also makes the spend path testable without a
running proxy, which the previous shape was not.
* refactor(spend): measure savings against one configured model, not a derived one
The counterfactual was derived per request: enumerate the hardest tier's
deployments, resolve each one's effective pricing, price them all, take the dearest.
That machinery produced a review finding per input shape it had not anticipated,
and every answer it gave was one an operator could have stated in a line of config.
So they state it. `litellm_settings.autorouter_savings_baseline_model` names the
model the traffic would have run on without a router, for every auto-router on the
proxy, and unset means the driver is off rather than a model nobody named being
guessed at. `savings_baseline.py` and its tests are deleted outright, along with the
tier enumeration, the candidate list on the routing decision, and the per-deployment
override that shadowed it.
Cache-state handling is untouched: the baseline is still priced on this request's own
read and write split, so a switch still pays for re-warming the cache and a first
turn still charges the write to both arms.
45 insertions against 482 deletions.
* refactor(router): compute the conversation shape once and pass it down
`_classify_and_route` re-derived it from the messages the hook had already resolved,
so an ordinary routed request walked the turn list twice for one boolean. The hook
computes it and hands it over, which is also where the affinity-hit path already got
it from.
Also moves `_get_llm_router` below the imports it sat among.
* fix(router): drop the dead conversation_continuing parameter off the hook
It was added to `async_pre_routing_hook` by mistake and immediately overwritten by
the value the hook computes, so it never did anything. It also widened a signature
every pre-routing strategy shares with the protocol in `types/router.py`, leaving
this one router diverged from `AutoRouter` and the interface for no reason.
Also records why an unreadable request counts as continuing: no messages is no
evidence a turn was served, so it pays the cache write and under-claims rather than
being handed a first turn's larger saving on nothing.
* fix(spend): charge a baseline its input rate for cache buckets it cannot price
A model with no cache_creation_input_token_cost, which is every OpenAI, Azure and Gemini entry, resolved that rate to 0.0 and carried the whole written prompt for free, so a first turn routed onto a cheaper model reported a loss. Same hole on cache reads. Those tokens are plain input on such a model, so they move into the text bucket.
* refactor(spend): build the daily upsert payloads in one shot
`common_data` and `update_data` were constructed and then appended to: `request_id`
conditionally for tag rows, `endpoint` unconditionally a few lines later. A dict that
grows after its literal cannot be reasoned about by reading the literal, which is the
whole point of building it at once.
The conditional key resolves to a spreadable value before either payload, so both are
single expressions and the tag branch appears once instead of twice.
Not wrapped in MappingProxyType, though it was suggested: these go straight to
prisma, whose query builder branches on `isinstance(value, dict)` to tell a nested
node from a scalar. A mappingproxy is a Mapping but not a dict, so it falls through
to the serializer and raises `TypeError: Type <class 'mappingproxy'> not
serializable` inside the batch upsert, where the surrounding except would log it and
leave the rollups silently unwritten.
* fix(spend): keep the one-shot upsert payloads under the type-discipline budget
Building both payloads as single literals traded a mutation for two dict literals,
and LIT002 counts construction rather than mutation, so the change the review asked
for is the one the gate charges for.
The empty branch is the avoidable half: it is the same value every time, so it moves
to a module constant built once instead of a literal per transaction, and it is a
read-only mapping so none of the call sites that spread it can fill it in later.
The block event Rubrik receives sourced caller identity from
model_call_details[metadata], where the enriched litellm metadata never
lives; it sits under litellm_params. Every block therefore reported
user_api_key_hash as an empty string, so a security block could not be
traced to a key, user, or team.
Read identity off the authenticated UserAPIKeyAuth the failure hook is
already handed, via the same mapper the success path and the proxy spend
logger use, so a block log and a success log describe their caller with an
identical key set.
"Add keyword rule" seeds a row with no keywords, and the only check that a
rule carried one lived inside getSemanticConfigError, which returns early
when semantic keyword matching is off. Off is the default, so an unfilled
row fell through to serializeKeywordTierRules and was discarded on the way
to the payload; the create reported success and the rule was gone.
The row now reports the gap itself and the submit is withheld while one is
outstanding, on the create form and the edit modal alike, both reading
emptyKeywordTierRuleIndexes so the row named and the row marked cannot
differ. Enter commits a typed keyword: the dropdown is kept closed, which
left antd nothing for Enter to select, and submitting was what used to
supply the blur that saved the word.
The backend already refused such a rule, but only when the router built the
deployment, so a caller that sent one anyway got the row written, dropped on
reload, and a 500. The management write paths now parse the incoming
complexity_router_config with the router's own ComplexityRouterConfig, judged
on the config alone so a patch that writes one without naming a model is
covered too, and reject it with a 400 having persisted nothing.
* fix(bedrock): stop forwarding no-op toolSpec.strict to Converse
`strict: false` is the Chat Completions default, so sending it to Bedrock
Converse communicates nothing the provider does not already assume, while
Bedrock rejects the key by presence rather than by value: any Claude model
routed through its Anthropic-compatible validator 400s with
`tools.0.custom.strict: Extra inputs are not permitted`.
The existing `bedrock_converse_supports_strict_tools` gate only protects
models whose `model_prices_and_context_window.json` entry carries the flag,
which makes every newly released Claude model broken by default until someone
adds it. That is a losing race for a field that carries no information when
false, and it is unrecoverable from the client side on `/v1/responses`, where
the Responses to Chat Completions bridge stamps `strict: false` onto every
function tool even when the caller never sent one. `drop_params` cannot help
there because the caller never supplied the param.
Drop the key when falsy instead. `strict: true` still honors the per-model
gate, so models that accept strict schemas keep the behavior they have today
and the flag keeps doing its job for the values that actually mean something.
* fix(bedrock): flag Claude Sonnet 5 as rejecting toolSpec.strict
Bedrock routes Sonnet 5 through the Anthropic-compatible validator that
rejects `toolSpec.strict`, but its six pricing-map entries never got
`bedrock_converse_supports_strict_tools: false`, so the gate fell back to
forwarding for Anthropic models and every tool call carrying `strict: true`
400'd. Verified live in us-east-1: before this, `strict: true` against
`us.anthropic.claude-sonnet-5` returns
`tools.0.custom.strict: Extra inputs are not permitted`; after, it returns a
real tool call.
Measured the rest of the family the same way rather than trusting the map:
Sonnet 4.5, Sonnet 4.6 and Haiku 4.5 all accept `strict: true`, and Opus 4.8
already carries the flag. Sonnet 5 was the only entry where the map disagreed
with the provider, so it is the only one changed here.
Same shape as the Opus 4.7/4.8 and Sonnet 4 fixes before it.
* ci(circleci): install a pinned Rust toolchain on the Linux jobs
The cimg/python images have no Rust toolchain, so every Linux job that
runs `uv sync` or `uv build` builds litellm-rust through maturin with no
cargo on PATH. maturin's puccinialin helper then fetches rustup-init from
the unversioned /rustup/dist/ path with no checksum and provisions a
floating `stable` toolchain, so the compiler a job builds with drifts
with whatever upstream published that day. uv hides build-backend output
on a successful sync, so none of this shows up in the job log.
Add an install_rust command that mirrors the Windows job: download a
pinned rustup 1.28.2, verify its SHA-256 against rust-lang's published
sidecar, install toolchain 1.97.1 with the minimal profile, and export
~/.cargo/bin through BASH_ENV. Run it after install_uv in every job that
builds the workspace; upload-coverage only runs `uv tool run coverage`
and is left alone.
Net download cost is unchanged, since puccinialin was already pulling a
rustup and a toolchain in each of these jobs.
* test(ci): guard that no CircleCI job builds the workspace without a pinned Rust
A green CI run does not notice the gap this closes: uv suppresses
build-backend output on a successful sync, so a job that syncs with no
cargo on PATH silently gets maturin's own unpinned rustup and a floating
toolchain, and the log looks identical either way.
Pin the invariant statically instead. Every job and reusable command is
walked in step order, and reaching a `uv sync` / `uv build` without a
Rust toolchain provisioned first is a failure. install_rust and the
Windows job's inline pinned install both satisfy it, so a new job that
forgets one is named in the assertion message at PR time. Separate cases
cover install_rust's own pins: a versioned /rustup/archive/ URL, a
SHA-256 verified before the installer is executed, and an exact
toolchain version rather than a channel name.
* ci(circleci): provision Rust for base_sdk_install
base_sdk_install landed on staging while this branch was open. It runs
`uv build --wheel` on cimg/python:3.12 behind install_uv alone, so it
built the bridge with maturin's own unpinned rustup. The guardrail added
here caught it on the merge result, which is the case it exists for.
* feat(team): custom metadata validation hook for team create and update
Operators can point general_settings.custom_team_metadata_validate at an
async Python function that validates team metadata before /team/new,
POST /team/update, and PATCH /team/{team_id} commit their writes. The
hook receives the metadata that will actually be written (the merged
result on PATCH) plus the stored metadata and requester context, and
fails closed: a rejected value returns the function's own message as a
400 while any exception or timeout blocks the write with a configurable
generic message as a 503. Premium-gated like enforced_params.
* fix(team): validate metadata before model alias writes and strip system keys from validator input
Review follow-ups on the team metadata validation hook: run the validator
before the model_aliases table insert so a rejected create leaves no
orphaned model rows, strip system-managed keys from existing_metadata so
the validator sees symmetric input on both fields, and accept class
instances exposing an async __call__ as validators. Adds a three-way
validator implementation matrix (allowlist function, HTTP-service-backed
function, immutability-enforcing class instance) driven through the real
create, update, and patch endpoints, including an HTTP stub service and
outage coverage.
* test(team): run the metadata validation matrix against the DB-backed proxy in CI
Adds the validator matrix to the proxy_store_model_in_db_tests CircleCI
job so every scenario runs full e2e against a Postgres-backed proxy. The
proxy config registers a dispatching validator that routes each request
to one of the three implementations via a metadata key and accepts
anything that does not opt in, keeping the rest of the suite unaffected.
CI starts a stand-in cost center service on the host for the HTTP-backed
implementation, reached from the container via host.docker.internal, and
the outage path targets a closed port to prove the fail-closed 503
without stopping services.
* feat(ui): edit team metadata as key-value pairs in team create and edit forms
The team create and edit forms asked for metadata as a raw JSON blob in a
textarea buried under Additional Settings. Both forms now render a key-value
pair editor directly under the TPM/RPM limit fields, backed by a shared
MetadataKeyValueFields component. Values round-trip losslessly: non-string
values display as JSON and parse back to their typed form on save, and
JSON-ambiguous strings are quoted so their type survives the trip. The edit
form hides UI-managed keys (logging, guardrails, model rate limits, etc.)
that dedicated controls already own and re-add on save.
* fix(ui): explain typed JSON parsing in the team metadata help text
* feat(team): schema-driven metadata fields from team_metadata_schema config
* refactor(team): render schema metadata fields as locked key-value rows, drop allowed_values
* refactor(team): schema fields reduce to key and label, tag-rendered keys, clean rejection toasts
* refactor(ui): prepopulate declared metadata keys as ordinary key-value rows
* fix(team): let non-admin dashboard users read the team metadata schema
* test(proxy): pin timeout wiring, boundary, and error-message contracts for team metadata validation
* fix(proxy): use pooled async httpx client in the e2e team metadata validator example
* refactor(team): satisfy staging lint ratchets inherited by the merge
* feat(guardrails/rubrik): prompt moderation, response-text blocking, streaming buffer, failure logging (#34019)
* feat(guardrails/rubrik): add prompt moderation, response-text blocking, streaming buffer, failure logging
- Add `pre_call` prompt moderation via `/v1/before_prompt/openai/v1` webhook:
structured messages are flattened and sent before the LLM is called; blocked
prompts surface a `ModifyResponseException` with the refusal text.
- Extend `post_call` response moderation to cover assistant text in addition to
tool calls; text blocks (wholesale replacement) are distinguished from
tool-block explanations (appended) via `startswith` diffing.
- Add `streaming_end_of_stream_only = True` and `streaming_buffer_until_moderated = True`
so streamed responses are withheld until end-of-stream moderation passes
(requires litellm >= BerriAI/litellm#31389; older versions fall back to
detect-only).
- Add `_MalformedToolBlockingResponseError` for structurally invalid service
responses; `_guarded` logs at CRITICAL so operators notice misconfiguration.
- Add `max_queue_size = 10_000`, `_enforce_max_queue_size`, and drop-oldest
backpressure so a webhook outage cannot grow the retry queue unboundedly.
- Add `flush_queue` override that snapshots once for both send and drain,
preventing duplicate delivery on concurrent flush calls.
- Make `_log_batch_to_rubrik` re-raise on error so `flush_queue` preserves
undelivered events for the next retry.
- Add `async_post_call_failure_hook` to log blocked requests
(`ModifyResponseException`) with a best-effort fallback payload for prompt
blocks (where no `standard_logging_object` exists yet).
- Add `_correlation_id` / `_apply_correlation_id` / `_prepend_system_prompt`
helpers; `_prepare_log_payload` now applies them for all providers (not just
Anthropic) so every log correlates by `litellm_call_id`.
- Add `get_supported_event_hooks` classmethod advertising `[pre_call, post_call]`.
- Use dedicated `httpx.AsyncClient` (`moderation_client`) for webhook calls
with explicit pool limits, separate from the shared logging client.
- Drop module-level `rubrik_handler` singleton (inappropriate for a library).
- Update `initialize_guardrail` docstring to explain `pre_call` vs `post_call` mode.
- Update tests: rename `tool_blocking_client` → `moderation_client`,
`tool_blocking_endpoint` → `response_moderation_endpoint`, `_flush_task` →
`_periodic_flush_task`; migrate `TestExtractBlockedTools` to
`TestExtractResponseBlock` for the new combined text+tool block API; add
tests for prompt moderation, text blocking, streaming flags, and failure
payload construction.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(guardrails/rubrik): add tests to reach 100% coverage
50 new tests across 18 classes covering previously-untested paths:
- Prompt moderation: passthrough, block, no-messages skip, message
flattening (content-list → string), payload construction with
tools/user/correlation_key/litellm_call_id fallback, refusal extraction
- async_post_call_failure_hook: non-matching exception no-op, missing
stash warning, valid stash → enqueue, AttributeError in payload build,
flush exception handling
- Block payload building: standard_logging_object present vs fallback
path, missing start_time
- async_log_success_event: _rubrik_blocked=True skip path
- aclose: task cancel + moderation_client.aclose()
- Edge cases: sampling rate clamp warning, unknown input_type passthrough,
empty-inputs early return, model_call_details warning, _stash_block_context,
duck-typed tool-call normalization, request_data["tools"] preference over
optional_params, system-prompt exception handler, flush-at-batch-size,
enqueue exception swallowing, queue empty/lock-None guards, non-dict JSON
response TypeError
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): use get_async_httpx_client, ruff format
- Replace bare httpx.AsyncClient with get_async_httpx_client (required
by ensure_async_clients_test; avoids per-request client creation)
- aclose() calls close() (AsyncHTTPHandler interface, not aclose())
- ruff format on rubrik.py and guardrail_hooks/rubrik/__init__.py
- Update 3 tests for AsyncHTTPHandler type (isinstance check, close())
osv-scan and documentation CI failures are pre-existing on the base
branch and unrelated to this PR.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): fix UP006 strict ruff violation
get_supported_event_hooks return type used List[...] (UP006) instead of
list[...]. Replace with the built-in generic and remove the now-unused
List import from typing.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): fix 3 reportArgumentType basedpyright violations
Use `# pyright: ignore[reportArgumentType]` (not `# type: ignore`) to
suppress the three errors basedpyright reports in --outputjson mode:
- convert_content_list_to_str call (dict vs AllMessageValues)
- _apply_correlation_id call (StandardLoggingPayload vs dict[str, Any])
- _prepend_system_prompt call (same)
Also tighten _apply_correlation_id and _prepend_system_prompt signatures
from bare `dict` to `dict[str, Any]`.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): don't close shared HTTP client in aclose()
moderation_client and async_httpx_client both come from LiteLLM's global
HTTP-client cache (get_async_httpx_client keys on llm_provider + params).
Two RubrikLogger instances with the same parameters share the same
underlying AsyncHTTPHandler object. Calling close() in aclose() closed
the shared connection pool for all instances, breaking any subsequent
moderation request on other loggers.
aclose() now only cancels the periodic flush task and lets LiteLLM
manage the shared client lifecycle. Tests updated to assert close() is
NOT called.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): use Counter for duplicate tool-call ID detection
Set-based comparison lost ID multiplicity: two original tool calls with
the same ID both appeared "allowed" even when the service returned only
one (e.g. one allowed + one prohibited sharing an ID). Replace with
Counter so returned_id_counts[id] >= required_id_counts[id] must hold
for every ID. Matches the approach in the original _extract_blocked_tools.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): respect default_on=true when omitted from config
LitellmParams.__init__ converts an omitted default_on to False before
initialize_guardrail receives it, so litellm_params.default_on is always
bool and never None. The is-None guard in RubrikLogger.__init__ therefore
never fired on the proxy path, leaving prompt/response moderation inactive
for any config that omitted default_on.
Fix: read the raw guardrail dict (before LitellmParams coercion) to
distinguish an explicit `default_on: false` from the absent-means-True
default. When the key is absent from the raw config, default_on=True is
used; when it is explicitly set (either True or False), that value wins.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: ruff format rubrik.py after Counter import addition
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): detect ID-less tool call removal; fix UP045
ID-less tool calls (tc.id is falsy) were excluded from required_id_counts,
so the Counter comparison never caught their removal. Add a cardinality
check (len(returned) < len(original)) that fires on any removal regardless
of ID presence, combined with the Counter check for duplicate-ID attacks.
Also fix 5 UP045 violations (Optional[X] → X | None) introduced by our
new code against the daily-branch baseline.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): filter optional_params through ModelParamHelper in fallback payload
_build_fallback_payload forwarded the raw optional_params dict as
model_parameters. optional_params can contain extra_headers, api_key,
and other upstream provider credentials that must not reach the Rubrik
webhook. The normal standard_logging_object path already filters through
ModelParamHelper.get_standard_logging_model_parameters(), which
allowlists only safe LLM API parameters. Apply the same filter here.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): scope failure hook by guardrail_name; moderate text-completions
Guard async_post_call_failure_hook by guardrail_name so multiple Rubrik
instances don't cross-log: the failure hook is called for every registered
callback; without the check the first instance pops the stash and the
originating instance finds None and silently skips logging. Now each
instance only handles blocks raised by itself.
Also moderate /v1/completions prompts: _moderate_prompt returned early
when structured_messages was absent. For text-completion requests litellm
supplies inputs["texts"] with no structured_messages. Added a fallback
that synthesises a user-message from texts so the before_prompt webhook
can evaluate text-completion prompts.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(lint): add reason comments to pyright: ignore suppressions
type-discipline budget requires each # pyright: ignore[...] to carry an
explanatory comment. Add reasons to the three bare suppressions on lines
483, 651, 652.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): include tool-call arguments in prompt moderation
_flatten_messages_for_moderation only sent the content field, silently
dropping tool_calls[].function.arguments and function_call.arguments.
An attacker could embed prohibited text in tool-call arguments inside
assistant history turns and bypass prompt moderation entirely.
Now collects all attacker-controlled text per message: text content via
convert_content_list_to_str, plus all tool_calls[].function.arguments
and the deprecated function_call.arguments, joined with newlines before
being sent to the before_prompt webhook.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): tighten append detection to prevent prefix bypass
startswith(sent_content) allowed any replacement whose text shares the
original as a prefix (e.g. "Hello" → "Hello, blocked.") to be classified
as a tool-block append rather than a text block, bypassing detection.
Use startswith(f"{sent_content}\n\n") to require the exact two-newline
separator the webhook uses between original text and appended tool-block
explanations. Also add `returned_content != sent_content` to text_blocked
so an unchanged passthrough is never classified as a block.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): default_on=False when omitted (follow existing pattern)
Remove the custom raw-dict lookup that was defaulting default_on to True
when omitted from the guardrail config. Follow the standard litellm
convention: omitted resolves to False (users must explicitly opt in with
default_on: true).
- initialize_guardrail: pass litellm_params.default_on directly
- RubrikLogger.__init__: is-None guard defaults to False not True
- Test updated to assert the correct False default
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore(rubrik): keep the ported guardrail within staging lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: credit the original author of the rubrik guardrail work
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: keep this mirror PR's diff limited to the rubrik files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): redact credential headers from request logging copies
clean_headers preserves an Anthropic subscription OAuth token, and other
client-supplied provider credentials, so they can be forwarded upstream. The
same dict was also stored as proxy_server_request["headers"] and
metadata["headers"], so those credentials reached every logging callback and
the SpendLogs proxy_server_request column that the Admin UI logs page renders.
Build the observability facing copies through redact_credential_headers, and
drop the transport-only keys (provider_specific_header, headers, api_key) from
the request body snapshot since they have to keep the real values.
* fix(proxy): use the redacted header copy in the request debug log
The stdout secret filter matches Bearer and sk- shaped values, so an MCP auth
token printed by the request-header debug line survived it in cleartext.
* fix(proxy): resolve the configured MCP auth header name through the secret manager
get_secret_str also consults a configured secret manager, so a deployment that
stores the header name there now gets that header masked too. Drops the added
comments in favour of a named constant.
* perf(proxy): resolve the MCP auth header name once per process
get_secret_str issues a blocking secret-manager SDK call when one is configured,
and configured_credential_header_names runs on every proxied request.
* fix(proxy): read the MCP auth header name live, cache only the secret manager
The config reloader rewrites os.environ on an interval and after /config/update,
and MCPRequestHandler resolves the same setting per request, so caching the env
lookup left a renamed header logged in the clear until the process restarted.
Only the blocking secret-manager call stays cached.
* refactor(proxy): narrow header redaction to the reported credential set
Drops the MCP header-name resolution, its per-request config and secret-manager
lookups, and the x-mcp- prefix rule. Those cover a separate credential family
than the one this ticket reports and carried their own config-reload staleness
surface; they belong in their own change.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Team-scoped DD credentials (dd_api_key, dd_site) set via POST /team/{id}/callback were silently dropped because _request_blocked_callback_params blocks them from standard_callback_dynamic_params. The security block is correct for request-level injection, but team callback_vars are admin-configured and trusted.
Store the raw init kwargs on the Logging instance and read dd_* params from there in _process_dynamic_callback_list instead of from standard_callback_dynamic_params.
Adds an integration test that exercises the full Logging.__init__ flow with team callback_vars to prevent regression.
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
* feat(ui): expose an Auto-Router session affinity toggle
session_affinity on ComplexityRouterConfig defaults to True, and neither the
create form nor the edit modal ever emitted the key, so every auto-router built
in the UI silently pinned each session to its first turn's model for an hour
with no way to see or change that.
Adds an "Advanced: Session Affinity" switch to both surfaces, defaulted on to
match the backend field. Both paths now write the key explicitly instead of
falling through to the backend default, so a stored config states what the
router actually does. A stored config with the key absent hydrates as on, since
those routers are running with affinity enabled today; showing them as off would
report the opposite of reality and persist it on the next save.
* feat(complexity_router): default session affinity off and expose it in the UI
session_affinity defaulted to True and the Auto-Router UI never emitted the
key, so every router built there silently pinned each session to whatever model
its first turn classified into for an hour, refreshed on every hit. There was
no way to see that from the UI and no way to change it without hand-editing
config.yaml.
The default flips to False, so every turn is classified on its own merits and
lands on the cheapest adequate tier. Pinning is now opt-in.
The toggle added in the previous commit follows the field: it renders off, and
both the create tab and the edit modal keep writing the key explicitly, so a
stored config states what the router does instead of inheriting a default that
can move under it.
Behavior change for existing routers: those created before this have no
session_affinity key stored, so they pick up the new default and start
reclassifying every turn. That gives up the provider prompt cache the pin was
preserving, and a multi-turn session can now change model between turns. Set
session_affinity: true to keep the old behavior.
Key and team `router_settings.model_group_alias` was accepted, persisted and
echoed back by `/key/info`, but never applied at request time, so the request
ran on the group the caller asked for. `route_request` forwards only the
settings the Router accepts as per-request kwargs, and `model_group_alias` is
not one of them: the Router resolves aliases from its own instance attribute,
which holds the global config map and is shared across requests.
Resolve the alias in the proxy instead, alongside the existing model-alias
rewrites and ahead of the pre-call hooks, so per-model limits and guardrails
key off the group that actually serves the request. Authorize the alias target
before the rewrite; model access was checked against the requested group, so a
key whose alias points at a group it cannot call gets the usual 403 rather than
being quietly served it.
Resolves LIT-4879
Pure rename, no behavior change. create_mcp_server.tsx and its test move
to CreateMCPServer, the two importers and one stale e2e comment follow,
and the local/filename-pascal-case suppression drops now that the file
passes the rule on its own.
The rename is scoped to this one component rather than the whole
directory because three PRs are currently open against its snake_case
siblings; the rest can follow once those land.
An evicted client was left for the garbage collector, but every OpenAI/Azure
SDK client is a reference cycle, so nothing freed the client or its pooled TCP
connections until a generational sweep ran. Driving 2000 azure calls through
the official image with no forced collection, live clients and open sockets
climbed from 202 to 1361 while the cache stayed at its 200-entry bound, and RSS
grew 279 MB to 456 MB against a TLS upstream.
Closing on eviction is what caused the earlier 'Cannot send a request, as the
client has been closed' regression, so an evicted client litellm created is now
closed only once a grace window has passed, by which point any request that was
already holding it has finished. A client the caller supplied is never closed,
since litellm does not own its lifecycle.
Resolves LIT-4883
* feat(teams): apply default organization to new teams from default team settings
Adds organization_id to DefaultTeamSSOParams so proxy admins can pick a
default organization in Default Team Settings. new_team applies it before
org validation whenever a team is created without an explicit
organization_id, so API, Admin UI, SCIM, SSO, and team upsert creations
all inherit it and go through the same existence and org-limit checks.
Explicit organization selections win and existing teams are untouched.
The default is validated at save time (PATCH /update/default_team_settings
returns 400 for an unknown org) and at create time, where a missing org now
surfaces as a clean 400 instead of a 500 by routing OrganizationNotFoundError
into the previously dead org_table None guard.
The Admin UI Default Team Settings tab gets a Default Organization row
backed by the shared OrganizationDropdown.
* fix(teams): validate org limits against final team state including defaults
Applies default_team_params and the legacy max_budget fallback before the
organization validation block, so _check_org_team_limits sees the values the
team will actually be persisted with. Also loads the org's budget table in
the lookup; without include_budget_table every budget comparison in
_check_org_team_limits was skipped because litellm_budget_table was None.
* test(proxy_behavior): pin org team limits as enforced on /team/new
The dead-code pins existed to turn red when include_budget_table went
live; that happened, so the scenarios now assert the 400 rejections plus
within-cap acceptance, and the unknown-org pin asserts the handler's 400
instead of the surfaced 500.
* fix(proxy): backfill null user_email on existing users during JWT auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): guard mapped-key email backfill and make null update atomic
Resolve Greptile review on the JWT user_email backfill:
- only backfill when the mapped virtual-key owner is the JWT principal, so a
mismatched admin-created mapping cannot write one user's email onto another
- make the best-effort mapped-key enrichment non-fatal so a database outage on
a cached-key request no longer fails otherwise-valid authentication
- persist the backfill with an atomic null-guarded update_many so concurrent
writers cannot overwrite an already-populated email
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep cache coherent when a concurrent backfill wins the null-email update
* fix(proxy): cache DB-persisted email after JWT backfill, not the proposed value
Resolve the Greptile finding that a successful null-guarded backfill could
cache this request's proposed email even if a concurrent ordinary user update
wrote a different email first. The helper now always re-reads the row after the
atomic update and refreshes the cache from the value the database holds, so
cache-hit auth and attribution stay consistent with the persisted record.
Annotate the Prisma and model_copy dict literals to keep the LIT002 budget within its ceiling.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
disable_team_logging cleared only metadata["callback_settings"], but callbacks
registered through POST /team/{team_id}/callback and the Admin UI live in
metadata["logging"], and request-time resolution stops at that slot without
ever reading callback_settings. The endpoint reported success while the team
kept sending request and response data to its third-party destination.
Empty the logging slot alongside the existing callback_settings reset, and
refresh the cached team object so the change applies to keys that are already
in flight rather than at the next cache expiry. The same refresh is added to
add_team_callbacks, which has the symmetric problem of a newly registered
callback staying dormant until the entry expires.
Resolves LIT-5101
The strict-priority e2e (added with the zero-increment limiter fix) can
never pass on stage: the proxy there does not run the
dynamic_rate_limiter_v3 callbacks + priority_reservation settings the
module requires, confirmed by zero limiter log lines across every
gateway and backend pod during the 2026-08-02 run. Config lives in the
infra repo; LIT-5118 tracks adding it.
The throughput SLO test failed the same run with 65.9% of requests dying
at the ELB as 502/503 before reaching a pod. The per-replica SLO rework
fixed the RPS-floor assertion but cannot help when stage idles at one
warm gateway replica; LIT-5119 tracks pre-scaling the fleet for the load
phase.
Both skips name their ticket, and the coverage registry returns the two
cells to the gap list while they are in place.
A single read of key_info.spend races the batched spend writer: deltas
earned before a reset flush to the DB up to ~60s later
(proxy_batch_write_at) and land on the row after the reset zeroed it.
The stage runs on Jul 30 and Aug 2 failed
test_key_budget_reset_at_advances_after_window exactly this way, with
spend back at the driven total while budget_reset_at had advanced and
calls flowed again.
Replace the single reads in rung 3 (spend zeroed after reset) and rung 4
(roomy window keeps spend) with _poll_key_spend, which re-reads to a 90s
deadline covering one full flush-plus-reset cycle. A reset that never
zeroes the row keeps spend pinned and still times out, so the regression
guard keeps its teeth.
* fix(team-callbacks): report API-registered callbacks from GET /team/{team_id}/callback
POST /team/{team_id}/callback writes metadata["logging"] while the GET read
metadata["callback_settings"], so every team configured through the API or the
Admin UI got back an empty list. c620d76fe4 migrated the writer to the new key
and left this reader on the old one.
Resolve the read the same way request-time resolution does in
_get_dynamic_logging_metadata: a logging slot that is present wins outright and
callback_settings stays as the deprecated fallback, so the endpoint reports what
a request would really do rather than the union of both shapes. An empty logging
list therefore reports no callbacks, matching a request that fires none.
Decrypt callback_vars for the response and mask the credential keys. Ciphertext
would be unusable to the caller, and a value encrypted under a key that is no
longer classified as sensitive would otherwise come back as a raw blob.
Resolves LIT-5093
* Update litellm/proxy/management_endpoints/team_callback_endpoints.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team-callbacks): mask callback vars that fail to decrypt
decrypt_callback_vars passes a value through untouched when it cannot be
decrypted, which happens to existing rows after a salt-key rotation. Under a
key that is not classified as sensitive that blob reached the caller as opaque
ciphertext it could not use or tell apart from a real value, so mask anything
still carrying the encrypted prefix.
Raised by Greptile on the first commit.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A keyless internal user signing in to the Admin UI was redirected off the
post-login landing to /ui/connect, which renders nothing but the MCP apps panel,
so a plain gateway sign-in ended on an MCP OAuth surface the user never asked
for. The landing now renders the keys dashboard for every role. The key lookup
that existed only to make that routing decision goes with it, along with the
useKeys enabled flag it was the sole caller of and the role-hydration hold that
guarded its one-frame dashboard flash
The gateway DCR consent flow moves the other way. Its /authorize handed the
browser to /ui/chat/integrations, whose layout hard-blocks when enable_chat_ui
is off, which is the default, and client-side redirects to /ui/ without the
query string; that destroys the connect_flow handle and strands the MCP client
until the 600s flow cookie expires. It now lands on /ui/connect, which reads
connect_flow and connect_client, mounts the consent banner and puts the apps
panel in connect mode. /ui/chat/integrations keeps its connect-mode handling
this release so flows sealed before the deploy still finish
Resolves LIT-5104
Resolves LIT-4911
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.
B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.
ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
Two changes to the classifier's system role, both narrowing it rather than adding to it
classifier_tier_rubric let an operator replace the tier definitions. It shipped in
#35471 alongside the assistant-turn context window, but the two answer different halves
of the same report and only the context window was asked for. The override carried a
composed prompt, an overridable and a non-overridable half, a blank-is-unset rule, a
length-warning validator and a pair of dashboard controls. All of it goes
The rubric then closes on one of two lines, chosen by classifier_context_window_size.
At 0 no conversation is quoted, so the line is the original one, byte for byte: a
deployment that sends no context is told to classify the current message and nothing
else, which is what it could see all along. Above 0 the turns are quoted, and the
original line told the model to disregard them, which is how a request whose difficulty
was established in an earlier turn came back SIMPLE on the word "yes". There the line
instead says to classify the current message using the quoted turns as context, and to
rate what a short reply approves rather than the reply
The choice keys on the window and not on classifier_context_include_assistant_turns.
Whether the quoted turns are the user's alone or include the assistant's replies does
not change what the model needs told, and whose turn is whose is already on the turns.
Keying it on the assistant toggle would put the default deployment back on the original
line, which is the configuration the report was raised against
Folds in #35508, which built the window-dependent framing on top of the override this
removes; that PR is closed in favour of this one
`import litellm` reaches litellm/integrations/otel/model/config.py via
litellm_core_utils/litellm_logging.py, so pydantic-settings is needed at import
time. It was declared only in the `proxy` extra, which left a plain
`pip install litellm` unimportable on every platform.
Adds tests/base_sdk_tests/check_base_sdk_install.py and a base_sdk_install
CircleCI job that builds the wheel, installs it into a clean venv with no extras,
and smoke-checks the import, a mock completion, a mock embedding, the bundled
pricing metadata and the token counter. The check is stdlib-only on purpose;
installing pytest into that venv would add packaging, pluggy and iniconfig and
could mask the class of undeclared dependency it exists to catch.
Previously the Windows job was the only one installing without extras, so this
class of break was caught by accident rather than by design.
Gating the mock testing request params behind
general_settings.dangerously_allow_mock_testing_request_params (#35423) turned
every fallback, retry and timeout drill in tests/test_fallbacks.py into a 400:
the build_and_test job mounts proxy_server_config.yaml, which never opted in.
Opt that config in. It is the config the CI proxy runs with, and the suite it
serves exists to drive synthetic failures.
Add a unit test that ties the two together: it scans the top-level tests/test_*.py
files build_and_test globs for gated param names and fails if the config they run
against has not opted in, so the next change to either side is caught in a fast
lint-tier job rather than a Docker E2E.
Adding a team member by a user_id with no user row is now proxy-admin-only,
so the /team/member_add authz matrix, which targeted a never-seeded user_id,
started 403ing every non-proxy-admin caller. Seed the member as a real user
row so the matrix reads _validate_team_member_add_permissions alone; leaving
it unseeded and relaxing the expectations to 403 would have left all 18 rows
green with that gate deleted outright.
Cover the new gate at the HTTP boundary, where only the helper was pinned
before: a team admin and an org admin both clear the permission check on the
same team and are still refused an unprovisioned user_id, with no user row
left behind. Pin the escape hatch that refusal names too, so closing the
email-invite path for non-proxy-admins cannot pass silently.
Promote the user seeder the member-info pins had kept private to conftest,
and reclaim invited users by their scratch-prefixed email, since an invite
allocates the user_id server-side.
_delete_deployment stopped returning a count of evictions in #35400 and now returns
the frozenset of ids the db and config still want, so a caller judging its own reload
can tell a deliberate eviction from a deployment that went missing. These two tests in
tests/local_testing were left comparing that frozenset against an int and have been
failing since; the directory is only referenced by .circleci/config.yml, which no
longer reports checks on PRs, so nothing caught them.
The eviction behavior itself is unchanged, so the fix is on the assertions: compare
against the expected id set, and pin the router's surviving ids so a mutation that
evicts the wrong deployment is caught rather than passing a bare length check.
The componentized images exec uvicorn directly, so ddtrace-run never wraps the
interpreter. USE_DDTRACE is not inert there; the proxy lifespan still runs
patch_all and litellm's own manual spans still emit. What never gets installed
is ddtrace's ASGI TraceMiddleware: starlette builds its middleware stack lazily
on the first __call__, which is the lifespan scope, so patching from inside the
lifespan body is already too late and no root request span is ever created.
Route both entrypoints through a shared docker/component_entrypoint.sh that
mirrors the monolith's prod_entrypoint.sh contract, including the
DD_TRACE_OPENAI_ENABLED=False export that keeps ddtrace's openai integration
from double-reporting calls litellm instruments itself.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
The migrations image ran `prisma migrate deploy` against a bake anchored in
$HOME with no node in the runtime stage, so prisma-client-py fell through to
nodeenv and tried to download a Node runtime on first start. In an
egress-restricted cluster that fails outright, and under an arbitrary uid the
uid-specific cache path is unreadable, so the job never applies a migration.
Move the bake to /opt/prisma with world-readable modes, install node in the
runtime stage, and pin PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH /
PRISMA_OFFLINE_MODE so the migration entrypoint runs the cached CLI directly.
This is the same treatment the root, non_root and database images already
carry.
Resolves LIT-4727
The floor was an absolute fleet number, so it asserted replicas x per-replica rate
and went red on how many gateway pods happened to be warm rather than on the request
path. The test now measures one replica first, with a short serial pass that only ever
occupies a single pod, and requires the concurrent phase to reach at least that rate.
A serial latency budget carries the request-path assertion the floor used to imply,
and both hold at one replica or seven.
Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load
model is a mock_response deployment with no upstream, a single-worker replica serves
it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at
6s each by Little's law.
The runner also kept locust's --json summary and threw away everything else, so a run
where 93% of requests failed said nothing about what they got. It now passes --csv,
reads the failure breakdown back, and reports locust's own generator-saturation
warnings, both folded into the assertion messages.
Resolves LIT-5054
The LLM classifier's context window carried user turns only, so a conversation
whose difficulty was stated by the model rather than by the user was classified
without it. Asked to find events, the assistant answers "here is the plan, it is
complex, should I execute?", the user answers "yes", and the router rates the
word "yes" and picks the cheapest tier
Two independent causes, so two changes that are each provable on their own
classifier_context_include_assistant_turns adds assistant turns to the window.
It is off by default because turning it on shifts tier decisions, and therefore
spend, for an already-deployed router, and because assistant text is net-new
egress to the classifier deployment. With it on, classifier_context_window_size
counts the last N turns across both roles, which is what makes the assistant's
own statement of difficulty land in the window
Assistant text reaches the classifier payload and nothing else. The window is
read only by _build_classifier_user_payload, while keyword_tier_rules, escalation
matching, the heuristic scorer and the semantic embedding all read the human ask
through _iter_human_asks_newest_first. Those are substring and vector matchers,
so an assistant echoing an escalation keyword back to a user would choose the
model, and the spend, with nobody having asked. Rather than widen the shared
iterator, _iter_context_turns_newest_first is separate and feeds the window
alone, which makes the boundary structural instead of a rule to remember
The rubric ended "Classify only the current message", and the classifier applied
it literally: a request whose difficulty was established earlier came back SIMPLE
because the message being rated was the word "yes". A context window the rubric
then tells the model to disregard buys nothing, so the wording now asks it to
rate the work the current message approves, judged in the conversation it
continues, while still forbidding it to rate a quoted section as if that section
were the request
classifier_tier_rubric lets an operator replace the tier definitions. The
trust-boundary paragraph is appended and cannot be replaced: it defends the
operator against their own callers, so an operator writing tiers without that
threat in mind would otherwise hand every keyholder the top tier by omission.
Blank reads as unset so an empty form field falls back rather than sending a
rubric with no tiers in it
Turns are labelled by role only when assistant turns can appear, so the prompt of
every deployment that never asked for this is unchanged byte for byte
The explicit AssumeRole branch of BaseAWSLLM.get_credentials returned without
touching the process-wide IAM cache, so every model request issued a fresh
sts:AssumeRole, and on ECS/EC2 an uncached sts:GetCallerIdentity ahead of it.
Route the whole role branch through _get_or_set_cached_credentials with the TTL
_auth_with_aws_role already computed and discarded. The cache key is the same
aws_* argument snapshot the other flows use, taken before the session-name
default is filled in, so each aws_session_name keeps its own STS session and no
attributed identity can be served another's credentials.
Credential fetches now single-flight behind striped locks. Without that, a burst
of concurrent misses on one key each issued their own STS call, which is the
same thundering herd the cache exists to prevent, moved to the miss window.
A daily-spend batch upsert that outlives prisma-client-py's 30s HTTP read
timeout keeps running server side after the client gives up, holding its
row locks for as long as the database takes. Every later flush cycle
queues behind those locks, which is how one slow batch cascaded into
exhausted database sessions.
The query engine's own transaction timeout cannot end that wait: it
cannot interrupt a statement that is already executing. Measured against
real Postgres, a batch wrapped in db.tx(timeout=60s) still held its locks
for the full 90s the statement ran. Only a Postgres-side statement_timeout
bounded it.
database_statement_timeout and database_lock_timeout (seconds) are now
first-class general_settings keys, emitted as libpq
options=-c statement_timeout=<ms> on DATABASE_URL. They are opt-in, so an
unset config keeps today's behavior, and they are never applied to
DIRECT_URL, which serves migrations that legitimately run long.
Resolves LIT-4718
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
The Prisma query engine is a separate Rust process whose resident memory is
a high-water mark: it grows with the payload of the largest single statement
it executes and glibc never returns that memory to the OS, so a pod's memory
floor ratchets up to its worst-ever write and stays there for the life of
the worker. Memory-based autoscaling then reads a number that reflects the
largest write the pod has ever done rather than what it is doing now.
The spend-log flush handed Prisma a fixed 1000 rows per create_many. With
store_prompts_in_spend_logs enabled a single row carries the full prompt and
response, so one statement can be tens of megabytes and permanently costs
hundreds of megabytes of RSS. Row counts cannot express that budget: the
same 1000 rows range from well under a megabyte to tens of megabytes.
Split each flush into statements bounded by encoded payload size
(SPEND_LOG_WRITE_BATCH_MAX_BYTES, default 2MB) on top of the existing
1000-row cap. What is measured is the encoded statement, so the budget
counts what actually goes on the wire: the JSON escaping of quotes and
newlines, multibyte characters at their encoded width, the field names and
separators a 25-column row carries, and the brackets and row separators the
rows carry as one collection. Deployments that do not store prompts keep one
statement per 1000 rows and are unaffected; prompt-carrying flushes get
several small statements instead of one huge one. A row larger than the
budget is still written on its own rather than dropped, and a row the
serializer refuses counts as zero rather than raising out of the flush and
dropping every row queued behind it.
Splitting a flush must not multiply what a poison-row flood costs, so the
poison-isolation allowance is threaded through every statement of a 1000-row
group instead of being handed out fresh per statement. That is only safe
because the allowance now counts failed inserts rather than every insert:
the one insert a statement needs when nothing is poisoned is not charged, so
a healthy flush never runs the allowance down however many statements it
splits into, and a statement reached after the allowance is spent is still
attempted so clean rows behind a flood still persist. Failed inserts for a
group are bounded by the allowance plus one baseline insert per statement,
which restores the constant-per-group ceiling the single-statement path had.
Resolves LIT-4765
