The classifier's system role was built from the rubric override alone, so the same
bytes shipped whether classifier_context_window_size was 0, 3 or 5. One static
string cannot describe both payloads, and it was wrong in both directions.
At 0 nothing about the conversation reaches the payload, yet the boundary paragraph
still promised quoted prior turns and asked the model to rate "the work the current
message asks for, judged in the context of the conversation it continues", which on
a bare "yes" is a demand to weigh an exchange it cannot see. Above 0 the window was
quoted but the model was never told it existed or that its view was bounded, so the
operator's choice of N never reached it.
The system role is now tier definitions, then the trust boundary, then a framing
sentence selected by the window: above 0 it names the configured count and keeps the
short-reply guidance, at 0 it says plainly that no earlier turns are quoted. Not a
revival of "Classify only the current message", which told the model to disregard
context that was present; this describes a payload that has none. The prior-turn half
of the boundary moved into that framing, since the caller's system prompt is quoted at
every setting and prior turns are not.
The framing keys on configuration and never on the individual request, so the system
role stays prompt-cacheable across a session's classifier calls; "may be quoted"
already covers a single-turn request with nothing to quote.
The componentized images exec uvicorn directly, so ddtrace-run never wraps the
interpreter. USE_DDTRACE is not inert there; the proxy lifespan still runs
patch_all and litellm's own manual spans still emit. What never gets installed
is ddtrace's ASGI TraceMiddleware: starlette builds its middleware stack lazily
on the first __call__, which is the lifespan scope, so patching from inside the
lifespan body is already too late and no root request span is ever created.
Route both entrypoints through a shared docker/component_entrypoint.sh that
mirrors the monolith's prod_entrypoint.sh contract, including the
DD_TRACE_OPENAI_ENABLED=False export that keeps ddtrace's openai integration
from double-reporting calls litellm instruments itself.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
The migrations image ran `prisma migrate deploy` against a bake anchored in
$HOME with no node in the runtime stage, so prisma-client-py fell through to
nodeenv and tried to download a Node runtime on first start. In an
egress-restricted cluster that fails outright, and under an arbitrary uid the
uid-specific cache path is unreadable, so the job never applies a migration.
Move the bake to /opt/prisma with world-readable modes, install node in the
runtime stage, and pin PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH /
PRISMA_OFFLINE_MODE so the migration entrypoint runs the cached CLI directly.
This is the same treatment the root, non_root and database images already
carry.
Resolves LIT-4727
The floor was an absolute fleet number, so it asserted replicas x per-replica rate
and went red on how many gateway pods happened to be warm rather than on the request
path. The test now measures one replica first, with a short serial pass that only ever
occupies a single pod, and requires the concurrent phase to reach at least that rate.
A serial latency budget carries the request-path assertion the floor used to imply,
and both hold at one replica or seven.
Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load
model is a mock_response deployment with no upstream, a single-worker replica serves
it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at
6s each by Little's law.
The runner also kept locust's --json summary and threw away everything else, so a run
where 93% of requests failed said nothing about what they got. It now passes --csv,
reads the failure breakdown back, and reports locust's own generator-saturation
warnings, both folded into the assertion messages.
Resolves LIT-5054
The LLM classifier's context window carried user turns only, so a conversation
whose difficulty was stated by the model rather than by the user was classified
without it. Asked to find events, the assistant answers "here is the plan, it is
complex, should I execute?", the user answers "yes", and the router rates the
word "yes" and picks the cheapest tier
Two independent causes, so two changes that are each provable on their own
classifier_context_include_assistant_turns adds assistant turns to the window.
It is off by default because turning it on shifts tier decisions, and therefore
spend, for an already-deployed router, and because assistant text is net-new
egress to the classifier deployment. With it on, classifier_context_window_size
counts the last N turns across both roles, which is what makes the assistant's
own statement of difficulty land in the window
Assistant text reaches the classifier payload and nothing else. The window is
read only by _build_classifier_user_payload, while keyword_tier_rules, escalation
matching, the heuristic scorer and the semantic embedding all read the human ask
through _iter_human_asks_newest_first. Those are substring and vector matchers,
so an assistant echoing an escalation keyword back to a user would choose the
model, and the spend, with nobody having asked. Rather than widen the shared
iterator, _iter_context_turns_newest_first is separate and feeds the window
alone, which makes the boundary structural instead of a rule to remember
The rubric ended "Classify only the current message", and the classifier applied
it literally: a request whose difficulty was established earlier came back SIMPLE
because the message being rated was the word "yes". A context window the rubric
then tells the model to disregard buys nothing, so the wording now asks it to
rate the work the current message approves, judged in the conversation it
continues, while still forbidding it to rate a quoted section as if that section
were the request
classifier_tier_rubric lets an operator replace the tier definitions. The
trust-boundary paragraph is appended and cannot be replaced: it defends the
operator against their own callers, so an operator writing tiers without that
threat in mind would otherwise hand every keyholder the top tier by omission.
Blank reads as unset so an empty form field falls back rather than sending a
rubric with no tiers in it
Turns are labelled by role only when assistant turns can appear, so the prompt of
every deployment that never asked for this is unchanged byte for byte
The explicit AssumeRole branch of BaseAWSLLM.get_credentials returned without
touching the process-wide IAM cache, so every model request issued a fresh
sts:AssumeRole, and on ECS/EC2 an uncached sts:GetCallerIdentity ahead of it.
Route the whole role branch through _get_or_set_cached_credentials with the TTL
_auth_with_aws_role already computed and discarded. The cache key is the same
aws_* argument snapshot the other flows use, taken before the session-name
default is filled in, so each aws_session_name keeps its own STS session and no
attributed identity can be served another's credentials.
Credential fetches now single-flight behind striped locks. Without that, a burst
of concurrent misses on one key each issued their own STS call, which is the
same thundering herd the cache exists to prevent, moved to the miss window.
A daily-spend batch upsert that outlives prisma-client-py's 30s HTTP read
timeout keeps running server side after the client gives up, holding its
row locks for as long as the database takes. Every later flush cycle
queues behind those locks, which is how one slow batch cascaded into
exhausted database sessions.
The query engine's own transaction timeout cannot end that wait: it
cannot interrupt a statement that is already executing. Measured against
real Postgres, a batch wrapped in db.tx(timeout=60s) still held its locks
for the full 90s the statement ran. Only a Postgres-side statement_timeout
bounded it.
database_statement_timeout and database_lock_timeout (seconds) are now
first-class general_settings keys, emitted as libpq
options=-c statement_timeout=<ms> on DATABASE_URL. They are opt-in, so an
unset config keeps today's behavior, and they are never applied to
DIRECT_URL, which serves migrations that legitimately run long.
Resolves LIT-4718
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
The Prisma query engine is a separate Rust process whose resident memory is
a high-water mark: it grows with the payload of the largest single statement
it executes and glibc never returns that memory to the OS, so a pod's memory
floor ratchets up to its worst-ever write and stays there for the life of
the worker. Memory-based autoscaling then reads a number that reflects the
largest write the pod has ever done rather than what it is doing now.
The spend-log flush handed Prisma a fixed 1000 rows per create_many. With
store_prompts_in_spend_logs enabled a single row carries the full prompt and
response, so one statement can be tens of megabytes and permanently costs
hundreds of megabytes of RSS. Row counts cannot express that budget: the
same 1000 rows range from well under a megabyte to tens of megabytes.
Split each flush into statements bounded by encoded payload size
(SPEND_LOG_WRITE_BATCH_MAX_BYTES, default 2MB) on top of the existing
1000-row cap. What is measured is the encoded statement, so the budget
counts what actually goes on the wire: the JSON escaping of quotes and
newlines, multibyte characters at their encoded width, the field names and
separators a 25-column row carries, and the brackets and row separators the
rows carry as one collection. Deployments that do not store prompts keep one
statement per 1000 rows and are unaffected; prompt-carrying flushes get
several small statements instead of one huge one. A row larger than the
budget is still written on its own rather than dropped, and a row the
serializer refuses counts as zero rather than raising out of the flush and
dropping every row queued behind it.
Splitting a flush must not multiply what a poison-row flood costs, so the
poison-isolation allowance is threaded through every statement of a 1000-row
group instead of being handed out fresh per statement. That is only safe
because the allowance now counts failed inserts rather than every insert:
the one insert a statement needs when nothing is poisoned is not charged, so
a healthy flush never runs the allowance down however many statements it
splits into, and a statement reached after the allowance is spent is still
attempted so clean rows behind a flood still persist. Failed inserts for a
group are bounded by the allowance plus one baseline insert per statement,
which restores the constant-per-group ceiling the single-statement path had.
Resolves LIT-4765
A failing deployment stamps its own litellm_params.num_retries onto the raised
exception, and async_function_with_retries adopted that value unconditionally. So a
model_list num_retries outranked both the x-litellm-num-retries header and the request
body, inverting the documented precedence to model_list > header > body >
litellm_settings.
The router could not tell a request-level value from its own default because the entry
points filled num_retries in with self.num_retries whenever the caller omitted it,
collapsing "the request asked for N" and "nobody asked". Drop that pre-fill from
_update_kwargs_before_fallbacks and from the six entry points that also did it a line
above their own call to it (image generation sync and async, adapter completion, file
create, batch create, batch cancel), all of which reach async_function_with_retries,
where the router/global default is already resolved. Leaving them would have made the
request value never None on those routes and permanently suppressed a deployment
num_retries there.
The sync text_completion pre-fill stays. That path resolves a deployment and calls
litellm.text_completion directly, never entering the retry loop, so no request-versus-
deployment ranking happens there and there is nothing to fix; removing the line would
only change which value is forwarded to litellm.text_completion, a behaviour change this
bug does not call for.
async_function_with_retries then adopts the deployment's value only when the request
carried none. Precedence is now header > body > model_list > litellm_settings, with the
deployment value still beating litellm_settings when the request is silent, on every
entry point that retries.
Resolves LIT-4772
The public A2A guide tells users to declare agents under a top-level
`agents:` key, but the proxy only ever read `agent_list:`, so the
documented config was silently ignored and GET /v1/agents returned an
empty list. Accept `agents` as the documented spelling and keep
`agent_list` working for anyone who found it by reading the source.
Selection is by key presence, so an explicitly empty `agents: []` is not
overridden by leftover legacy entries.
Config-defined agents were also dropped on any database-backed gateway:
the periodic reload rebuilt the registry from the DB rows plus a module
global that was declared and never assigned. The registry now remembers
the agents it loaded from config.yaml and replays them on every rebuild.
A database row wins a name collision, mirroring how config-declared MCP
servers are unioned under the database registry, so name lookups and
deregistration keep addressing exactly one agent.
Resolves LIT-4978
Aligns the fix with the constraints in LIT-4800. A zero-increment
counter now blocks at current >= limit, matching RPM's semantics; the
previous current > limit let a pool sitting exactly at its reservation
admit one extra request. reserve_tpm_tokens rebuilds its descriptors
with only tokens_per_unit so the requests dimension stays out of the
reservation pass, which deliberately leaves RPM to the separate
should_rate_limit check.
* feat(prometheus): add global exclude_metrics and exclude_labels options
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): apply global exclude_labels to hard-coded metric labels
Metrics built with hard-coded labelnames lists (guardrail, provider budget,
callback, managed file/batch, batch cost) bypassed prometheus_exclude_labels
because only labels resolved via get_labels_for_metric were filtered. Route
every metric through a factory that strips excluded labels at construction and
proxies labels() so excluded labels are dropped at emission too. Add the
non-enum hard-coded labels (guardrail_name, status, error_type, hook_type,
purpose, file_type, result) to exclude-config validation so they are accepted
instead of raising ValueError at logger init.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(prometheus): simplify exclude-label factory to the kwargs labelnames path
All metric definitions pass labelnames as a keyword argument, and the only
metrics that pass it positionally resolve their labels through
get_labels_for_metric, which already drops excluded labels, so they never carry
an excluded label into the factory. Drop the unreachable positional
reconstruction branch.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: re-trigger CI (flaky unrelated bedrock agentcore test)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): use immutable constructions to satisfy LIT002 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keys created with key_type=llm_api get allowed_routes=["llm_api_routes"],
which covered /v1/models but not /v1/model/info, so a client could list model
names but not read pricing, mode, or max_tokens without a second key.
Adds both /model/info and /v1/model/info (same handler) to llm_api_routes only.
Membership there is not the same as RouteChecks.is_llm_api_route(), which is
what gates DISABLE_LLM_API_ENDPOINTS, global/virtual-key budget enforcement,
enforce_user_param and JWT team attachment; /guardrails/apply_guardrail already
sits in the group the same way. /v2/model/info stays out: it is the paginated
Admin UI listing, not model metadata a caller needs at request time.
public_routes moves from set([...]) to a frozenset literal to keep the LIT002
and ruff-strict ceilings from rising; both budgets ratchet down by one.
add_deployment already reapplies DB router settings through _update_llm_router,
so gating router_settings out of the pub/sub publish set left the push path
covering less than the resync actually applies
Resolve the requested member user_ids with a single find_many instead of one
lookup per member, so a large member list no longer turns into that many
round-trips before the permission check runs. Write the member-add audit
entries concurrently rather than one after another, and list at most a few
ids in the rejection message instead of echoing the whole request back.
Update the team-admin member-add case that covered adding a user_id with no
user row, which the endpoint now leaves to proxy admins.
Caps fleet-wide reload rate at one resync per 10s per pod so a burst of
authenticated writes cannot amplify into continuous cross-pod reloads, and
skips publishing config params (environment_variables, router_settings) that
no resync callback applies outside proxy startup
After any management write to a DB-backed config table, publish an
invalidation event on the coordination Redis; every pod runs a
subscriber that debounces, jitters, and triggers an immediate
add_deployment plus get_credentials resync. The interval polls stay
as slow reconciliation fallback and behavior without Redis is
unchanged since publish and subscribe both no-op.
Adding a team member by a user_id with no user row created that row as a
side effect for any caller permitted to add members, while creating users
directly is restricted to proxy admins. Restrict that path to proxy admins
too; adding an existing user, and inviting a new one by user_email (where
the user_id is allocated server-side), are unchanged.
Also record the membership change, and any user row it creates, in the
audit log, matching /team/update, /user/new and /key/*.
* fix(ui): show pass through route selections in team/key forms and match team id substrings in team search
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(teams): keep team id search index-friendly with a prefix match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep /v2/team/list search id matching exact by default and add an opt-in prefix mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop model writes 500ing on another pod's delete
A model write judges the reload it triggers by diffing this pod's router before and
after, and reports anything that stopped serving as damage. On a pod that has not yet
polled a delete another pod made, the snapshot still lists that model; the reload then
evicts it because the db no longer has it, and the guard reads its own correct
reconcile as degradation. The row is written and served, but the caller gets a 500.
Since propagation between pods is a 30s db poll, any delete followed by a create
inside that window can land on a pod that has not caught up, so a delete-then-create
pair returns 500 whenever the two requests hit different pods.
_delete_deployment already computes exactly the set that settles it: the ids the db
and config still want. Thread it up through _update_llm_router, add_deployment and
clear_cache to the verdict, and intersect the drop set with it so an id the db no
longer has stops counting as collateral. Where no reconcile ran the set is None and
every drop is still reported, so a genuinely broken reload is caught as before.
_delete_deployment now returns that set instead of a delete count; the count had no
callers in the proxy, and the tests asserting it already assert the eviction calls.
* test(proxy): fold reload-verdict test commentary into docstrings and assertions
Greptile flagged the inline comments against the repo's no-new-comments rule. The
case-by-case context moves into the test docstring, and the two return-contract
assertions carry their reasoning as failure messages instead.
* test: fix clear_cache mock return type in model block/unblock tests
Review feedback: the relaxed predicate admitted negative increments,
which both atomic backends would apply as decrements. Restrict the new
behavior to zero-valued pure checks and assert negatives neither check
nor mutate counters.
The atomic check-and-increment path skipped any counter whose increment
was <= 0. The dynamic rate limiter always passes a zero token increment
pre-call because usage lands on the counters post-response, so on a model
configured with only tpm the limiter evaluated no counters at all: no
model-wide TPM cap and no priority reservation, in either generous or
strict mode. Regressed in dd57ae6691 when the pre-call flow moved off the
read-only should_rate_limit check, which did evaluate token limits.
Keep zero-increment counters in the payload so they act as a pure check
(current + 0 > limit), matching the pre-regression semantics in both the
Lua and in-memory paths. Adds unit regressions at the primitive and hook
level plus a live e2e covering the priority_generous/priority_strict
registry rows.
Handling of the client-supplied mock testing params was split across three
places with different behavior for each. Three were dropped from every proxy
request, two reached the router untouched, and a request that asked for a
synthetic failure came back as an ordinary success with nothing to indicate
that no failure had been injected
Put all six behind one opt-in, general_settings.
dangerously_allow_mock_testing_request_params, and reject rather than drop
when it is unset, so a fallback drill cannot report a pass for a test that
never ran. The rejection names the params it saw and the config key to set,
which is also the answer for anyone following the older docs
The flag is config-file only. It is deliberately absent from
ConfigGeneralSettings, and that absence is what makes /config/update drop it
on parse and /config/field/update reject it; the tests pin both so the field
cannot be added back for tidiness without the reason surfacing. Enabling it
logs a startup warning naming every param it unlocks
BREAKING CHANGE: mock_timeout and mock_testing_rate_limit_error now require
general_settings.dangerously_allow_mock_testing_request_params to be set in
config.yaml. Previously they were accepted unconditionally
Tool-level MCP entitlements are enforced in one place,
check_tool_permission_for_key_team, reached from pre_call_tool_check. Two
dispatch paths reached a tool handler without passing through it.
execute_mcp_tool's legacy fallback dispatched into the local tool registry
after retrying the unprefixed name, with no allowed/banned-tool check, no
key/team/org tool permissions and no parameter validation. It now runs the same
gate, and only when something can actually dispatch: when the unprefixed name is
absent from the local registry too, the existing 404 stands rather than becoming
a misleading "server unavailable".
The server the tool-level checks need is available even though the tool name is
not in the tool -> server mapping: a non-empty prefix has already been compared
against the caller's allowed_mcp_servers by exact name, so the named server is
in that list. It is resolved from allowed_mcp_servers rather than from the
manager's registry, because the registry can return a server the caller holds no
grant for, and matching on anything other than name would accept a server the
server-level check never validated. The remaining case is a prefix segment that
is empty, which the server-level check skips entirely because it is gated on a
non-empty server name; that now fails closed with 503 instead of dispatching for
a caller holding no server grant at all.
An entitled caller's legacy call therefore still dispatches, so a configuration
that worked before keeps working; only the unentitled call is refused, now with
the entitlement gate's own 403.
call_tool ran pre_call_tool_check inside `if proxy_logging_obj:`, so an absent
logging object would have skipped authorization silently. This half is defensive
with no live hole: all four call sites source the module-level ProxyLogging
singleton from proxy_server.py, which is never None. The shape was still wrong.
pre_call_tool_check now runs its three authorization checks unconditionally and
only the guardrail hooks, which are dispatched through the logger, depend on one
being present.
A third reported path, where allow_all_keys, BYOM-submitted and
upstream-delegated servers are unioned in after the resolver's ceilings, was
investigated and found not to be a defect. The widening is real, but a server's
tool surface is already boundable for every caller at registration through
MCPServer.allowed_tools / disallowed_tools, enforced by
check_allowed_or_banned_tools ahead of the entitlement check, and per-caller
narrowing plus the org tool ceiling remain available. Nothing here changes that
path.
Resolves LIT-4956
CheckBatchCost built unified output file ids with the provider model name, so key model-access checks resolved the file to e.g. gpt-5.5 and every GET /v1/files/{output_file_id}/content failed. Resolve the model group from the batch's managed input file, falling back to the deployment's model_name.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The hook resolves the newly created user through UserRepository and builds the
audit entry from that row. Pin both halves: the entry carries the persisted
row's fields rather than the /user/new response, and a user id that resolves to
nothing produces no entry at all.
* feat(spend-logs): record when a spend log row is the auto-router's own classifier call
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
* feat(ui): show which log rows are the auto-router's own classifier calls
A classifier row now carries internal_call_origin and shares its parent's session,
so the session trace lists it beside the request that triggered it. Without a marker
in the sidebar it reads as another call the caller made, which is the confusion this
resolves.
The tag renders only for a recognized origin, so ordinary traffic and any future
origin this build does not know about stay unlabelled rather than being asserted as
classifier calls.
* feat(proxy): add a generic list contract for management/v1 entity lists
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
* feat(proxy): add GET /management/v1/budgets
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
* fix(proxy): rework the budgets list onto the merged list contract
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
* refactor(proxy): fold the predicate renderer instead of recursing
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
* fix(tool-management): drop unsupported prisma select kwarg from team lookup
POST /v1/tool/policy returned HTTP 500 for every request carrying a team_id,
with "LiteLLM_TeamTableActions.find_unique() got an unexpected keyword argument
'select'". _resolve_team_id_to_object_permission_id looked the team up with
select={"object_permission_id": True}; prisma-client-py 0.11.0 has no select
kwarg on find_unique, so the call raised TypeError and the handler's except
block turned it into a 500. Both the initial read and the fallback read taken
when a concurrent writer wins the update_many race were failing the same way,
so per-team tool blocking was unreachable
The kwarg is dropped rather than replaced; the generated client has no
projection API, and this is a single-row lookup where selecting all columns
costs nothing worth working around
The existing tests missed this because the team table was an AsyncMock, which
accepts any keyword. The new double binds each call against the real
find_unique and update_many signatures, so an unsupported kwarg raises the same
TypeError production does
* refactor(test): tighten typing on the tool policy team table double
Replaces the double's Any annotations and bare list types with concrete ones:
kwargs are object, rows are Sequence[MagicMock] held as a tuple, and the call
logs are list[dict[str, object]]. Behaviour is unchanged; the double still
binds every call against the real generated prisma action signature, verified
by reintroducing the select kwarg and watching both regression tests fail
A bedrock guardrail configured mode: during_mcp_call never ran. ProxyLogging
remapped the event to during_mcp_call and dispatched, but bedrock's own
async_moderation_hook then hard-coded during_call and re-checked, so the second
check rejected the very requests the guardrail was configured for and the tool
call proceeded unscanned with no error.
Remap call_mcp_tool the way model_armor already does, which matches the remap
ProxyLogging.during_call_hook itself performs, and teach the shared
get_guardrails_messages_for_call_type helper that an MCP tool call carries its
payload in the same messages key, without which the hook passes the gate and
then bails on an empty message list.
The SLO measures how many gateway replicas happen to be warm rather than the
request path. Clearing the floor needs roughly 5-7 replicas at ~10-14 RPS each;
stage idles at one and reactive HPA scale-up lands minutes into a ~3 minute
test.
It failed both of its assertions on consecutive days: 93.3% errors at an
inflated 264 RPS, where the failing requests never reached a pod and closed-loop
RPS rose because they failed fast, then 16.7 RPS with zero failures.
The covers marker and registry row stay put, so the cell returns to the gap
list rather than disappearing from the denominator.
The ID-JAG egress arm could only assert a caller that presented its own IdP
identity token on the request, so an agent holding a brokered LiteLLM
credential got a 412 and never reached the upstream. The assertion captured at
SSO login was already persisted per user for exactly this purpose, but nothing
read it back.
The arm now falls back to that stored assertion, keyed on the authenticated
principal's user_id. The identity is always taken from the credential the
gateway authenticated, never from a caller-supplied field, so no caller can
select whose identity is asserted upstream. A missing, expired, or
unidentified subject stays a 412; ID-JAG exists to assert a specific user and a
missing subject has no safe substitute. A store outage is the one exception: it
is surfaced as a typed AssertionStoreUnavailable and mapped to 503, so a
database blip cannot 500 the egress or the upstream-401 retry, and does not
tell the user to sign in again over something they cannot fix.
Sourcing a subject from the store rather than the request changed what
invalidation can rely on, so the exchanged-token cache changed with it. The
entry is now addressed by a slot key derived from the principal, plus the
caller's own token when it presented one, with a fingerprint of the subject
token and config stored beside the bearer and compared on every read. A
mismatch reads as a miss and re-mints, so a rotated assertion or an edited
server config cannot be served a bearer authorized under the old inputs, and
two callers cannot receive each other's. Invalidation is a single delete of a
key it can always compute, needing no store lookup on the recovery path.
The upstream-401 invalidate-and-retry path was also gated on a truthy inbound
subject token, which skipped recovery entirely for store-sourced calls. The
gate is now mode-aware: token_exchange still requires an inbound token because
it has nothing else to mint from, id_jag does not.
oauth2_id_jag is also now selectable in the admin dashboard with its own field
set, instead of being reachable only from config.yaml or the REST API. The
auth-type selects drop antd list virtualization: at eleven options the last one
no longer mounts, which is a scroll in a browser but makes the option
unreachable to anything reading the rendered list.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
All three send a `telemetry` object in the arguments to Datadog's
search_datadog_logs tool. Datadog tightened that tool's input schema to reject
unknown properties, so every call now fails validation with 'unexpected
additional properties ["telemetry"]' before the behavior each test exists to
prove is reached.
`telemetry` was never a documented Datadog parameter; the tests relied on the
server ignoring extra properties. The proxy transmitted exactly what the tests
supplied and surfaced the upstream error faithfully, so this is test-side.
The covers markers and registry rows stay put: the collector counts a cell as
covered only when a test pytest would actually run declares it, so skipping
hands all four cells back to the gap list where they belong.
Replaces the double's Any annotations and List/Dict aliases with concrete
types, matching the equivalent double in the tool policy tests: kwargs are
object, records are Sequence[Mock] held as a tuple, and the call log is
list[dict[str, object]]. Behaviour is unchanged; the double still binds every
call against the real generated prisma action signature, verified by
reintroducing the select kwarg and watching the regression tests fail
* fix(aiohttp): dispose recycled client sessions deterministically
LiteLLMAiohttpTransport replaced its cached aiohttp.ClientSession on
loop-mismatch, loop-inspection failure, and "Session is closed" retry
without reliably closing the previous session:
- the close task from asyncio.create_task() was never referenced, so
it could be garbage-collected before running;
- the (RuntimeError, AttributeError) fallback branch replaced the
session without closing it at all;
- sessions bound to a closed event loop were abandoned to the GC
("rely on GC"), and sessions bound to a loop running in another
thread were closed from the wrong loop.
Replaced sessions surfaced as intermittent "Unclosed client session" /
"Unclosed connector" errors from the event-loop exception handler at
GC time.
_close_recycled_session() now covers the three lifecycles a recycled
session can be in: same-loop closes keep a strong task reference until
completion; sessions owned by a loop running elsewhere are closed on
their own loop via run_coroutine_threadsafe; sessions whose loop is
gone are disposed synchronously through the connector teardown that
aiohttp's own finalizer uses, which releases pooled connections and
silences the finalizer warnings.
Fixes#24230
* fix(aiohttp): guard threadsafe close callback against cancelled futures
---------
Co-authored-by: Anmol Jaiswal <68013660+anmolg1997@users.noreply.github.com>
