365 commits

Author	SHA1	Message	Date
Kazuki Hashimoto	c8ed5ccca8	fix(router): unregister strategy selectors from global callbacks on discard (#45795) ... * fix(router): unregister strategy selectors from global callbacks on discard Router.discard only removed its own bound methods from litellm's callback lists, so the strategy selectors it registered (least-busy, usage-based, latency-based, cost-based and routing group selectors) stayed behind and kept receiving events after the router was gone. Reuse the identity based unregister that update_settings already uses so a discarded router takes only its own selectors with it * fix(router): purge discarded selectors from every event callback list function_setup copies litellm.callbacks into the success and failure lists on every request, so a router that served traffic still leaked its selectors there after discard. Route the cleanup through remove_callback_from_all_lists, which now also covers input_callback, and fold the selector gathering into discard so the router coverage check sees no untested helper * fix(router): keep live routers, in-flight hooks and user_config limits intact on discard discard() now rebuilds every global callback list without the router's selectors, optional pre-call checks and own usage callbacks instead of removing them in place, so a callback loop already running never skips a hook. It then re-registers what the remaining live routers need, so a same-strategy survivor keeps its deduped handler. The proxy's per-request user_config router is discarded only after its call returns. * fix(router): bind user_config handlers to the request and tighten discard When a user_config request carries a logging object, its router is discarded before the request runs and its handlers ride on that logging object, so v1 rpm and v2 tpm limits count every request and concurrent requests count once. Without one, the router is discarded after the request as before. The router's pre-call loops also read the request-scoped handlers. discard() now refills only the handler keys it vacated, guards the callback lists and the live-router set with a lock, covers routers built without a model_list, and drops its Slack alerting, adaptive router hook and budget limiter sync task * fix(proxy): reuse one user_config router per config and bind its handlers to each request Every user_config request now goes through a cached router built once per config. Its strategy handlers leave the global callback lists and are bound to the request's logging object instead, so limits count each request once and nothing is left behind after the request. * fix(router): keep the proxy budget limiter on user_config requests A user_config router now swaps in only its strategy selectors, so the proxy's RouterBudgetLimiting stays on the request and keeps counting its spend. Usage logged after a strategy is retired, such as a request still in flight when its user_config router is evicted, now schedules one more push to Redis. * fix(proxy): init the retire fields on the key model budget limiter The key model budget limiter builds its own state instead of calling the base init, so with the proxy's Redis attached every spend increment raised on the missing retire flag and the key's model budget never counted spend. * fix(router): bind the deployment callbacks to user_config requests A cached user_config router serves requests through each request's logging object but bound only its selectors and policies there, so a failing deployment was never cooled down and its usage counters missed requests. * fix(router): retry an evicted router's budget writes and keep the adaptive header on user_config requests A retired budget limiter now keeps retrying its queued Redis increments every sync interval until Redis accepts them, instead of giving up after one failed write. The x-litellm-adaptive-router-model header now comes from one stateless hook in litellm.callbacks that holds no router, so user_config adaptive routers, whose per-router hooks are request-scoped, still answer with the header. * fix(router): lock request callback setup against router cleanup and drain in-flight budget writes on retire function_setup now edits the global callback lists under the same lock that router cleanup uses, so a router dropping its callbacks on another thread can no longer shift the indexes function_setup pops or put a dropped handler back. A budget limiter retired while a Redis write is still in flight now keeps retrying that batch if the write fails, instead of leaving the spend queued with no task left to send it. * fix(utils): set up loggers after releasing the callback lists lock function_setup held the callback lists lock while set_callbacks ran its one-time logger setup, which can block on a Slack package install, so other threads and Router.discard waited on it. The lock now covers only the list reads and writes. * test(router): annotate the retire test's locals as Final * fix(logging): skip replaced global callbacks on the input path too replace_global_callback already kept a same-key global logger out of success and failure dispatch. pre_call and post_call still sent it the input events, so a logger with an input hook could count a request start it never saw finish. Both now share one filter with the success path * test(logging): type the input-path regression test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-21 02:39:36 +00:00
Mateo Wang	e333ccabdb	fix(scim)!: delete a group through the team delete's bulk path instead of one member at a time (#47544) ... * fix(scim): write a group's roster in one locked transaction instead of one member at a time POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the membership rows and the users' team tags in one transaction under the team's advisory lock, so a 500-member push runs a fixed number of statements instead of thousands. PUT now answers with the roster it just wrote * test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers * fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock * fix(scim): evict the cached team on a group rename and read a no-op roster from the writer A rename-only PUT or PATCH wrote the new alias and left the team cached under its id and old alias on every worker, so keys kept logging spend against the old team alias until the TTL expired. The group write now evicts the cached team (and broadcasts the eviction) right after the alias or metadata update. sync_team_roster reads the team from the writer when the plan names nobody, so a replica cannot hand back the pre-rename row; a plan already matching the roster returns inside the lock without writing, evicting, or emitting the members metric; the response is built from the written row, so lastModified reflects the roster write; and detaching a removed member bumps the user's updated_at like every other user write. * fix(scim): bump detached members' updated_at in UTC The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts into Prisma's naive timestamp column in the session timezone, so on a database whose session is not UTC the removed member's row landed hours away from the UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and have the bulk roster cell assert the bumped row sits between two reads of the database's own UTC clock taken around the replace. * fix(scim): delete a group through the team delete's bulk path instead of one member at a time * fix(scim): chunk the group write's member lookups under the bind cap Every Prisma "in" filter the group write issues bound one parameter per member with nothing bounding the list, which check_unbounded_in_lists reports as 11 new findings. The single-field lookups now go through litellm.repositories.chunked_in (caller_transaction inside the roster transaction, per_chunk_ok for the role recompute), and the one cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two slices, and a member named only in the last slice still resolves. * fix(scim): demote a deleted admin group's members before the team delete so a failed delete stays retryable * fix(scim): recompute member roles again after the team delete so a concurrent group write cannot regrant admin * fix(proxy): send bulk cache evictions in slices so the broadcaster drops none Veria pointed out that the invalidation publisher caps its pending tasks at 1,024 and drops the excess with a warning, and that a group push now detaches every removed member's keys in one call instead of one member at a time, so a removal of more than roughly 340 members (three cache keys each) silently left stale key objects on the other replicas. The merge base never hit the cap because each member was a separate request. Bulk evictions (team, user, organization, and session key deletes, plus the roster settle) now publish in slices of 256 and wait, bounded at two seconds, for the backlog already handed to the event loop before each slice. A Redis that never answers costs a bulk caller at most that wait per slice instead of wedging the request, and single-key callers are untouched. Regression tests push 1,500 keys and a 500-member removal through a slow publisher and assert every eviction reaches Redis; both fail against the previous tip. * fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts Greptile pointed out that pacing the broadcasts in slices also paced the local eviction: a bulk delete waited for the publish backlog before dropping the next slice from this worker's own memory, so on a Redis that accepts connections but never answers a key deleted from the database kept authenticating on the handling worker for up to two seconds per slice. The merge base cleared every key locally at once. Bulk evictions now drop every affected entry from this worker's memory first, through the cache's own partition lookup since key objects live apart from the other auth objects, and only then publish in paced slices. The roster settle does the same for the deleted tokens, the JWT mapping keys and every changed member's user and membership entries before any step that can wait. Regression tests run both paths against a publisher that never answers and check that nothing is still cached on this worker before the first bound elapses; both fail with the local eviction removed. * fix(scim): write recomputed member roles only where the membership still backs them * fix(team): report the users a team delete detaches so SCIM demotes late-added members * test(scim): audit cells for the bulk group delete Move the SCIM group helpers into tests/integration/_support/scim.py and add the delete cells: a member's other group survives, the admin group's members are demoted and the audit row names the SCIM caller, a member a concurrent PUT adds while the delete waits on the team lock is demoted too, unknown, repeated, non-admin and hostile ids, an empty group, the peer proxy, a 20-delete burst across both proxies, and a delete after a worker was SIGKILLed * fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts * docs(proxy): drop the module docstring that repeated the PR description * fix(team): sweep deleted team references only under the lock so a late-added member is demoted The unlocked sweep that ran before the team's advisory lock stripped a member a concurrent group write added after the roster was read, so neither the roster snapshot nor the locked sweep's RETURNING named them and the SCIM recompute left them proxy_admin after the admin group was gone. The locked sweep does the same strip and reports whom it stripped, so it is the only sweep now. The integration cell holds the delete on a member's user row so a PATCH lands between the roster read and the lock, then checks the late member is demoted. Both waiter queries in the SCIM support module return their count as text so the first poll parses. * chore(team): carry the unbounded-IN baseline entries to delete_validated_teams The delete pass moved from delete_team and the unlocked sweep into delete_validated_teams, so the checker's baseline names the function the two team_id IN lists now live in * test(scim): expect the master key alias in the audit row and send post-kill requests on a fresh connection * fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps Bulk key evictions now drop entries from memory, delete them from Redis in one chunked DEL, and broadcast every key in one pipelined publish, so a large roster removal no longer overflows the publisher's pending cap or costs a Redis round trip per member. A push waits for in-flight membership loads before evicting. User rows are locked in byte order to match the spend writer, a bulk user delete takes every team lock before its first user write, and a POST whose roster write fails deletes the empty team so the IdP retry creates it again. * test(team): observe delete broadcasts at the Redis layer * fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently * fix(team): chunk the email lookup behind the member row lock * fix(scim): create a member deleted mid-push again instead of failing the group write * fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch A member deleted mid-push is created again only when scim_upsert_user allows it, under the verb's own created_via. New group members are checked against the license seat limit as one batch before any is created. Auth cache publish drops log the key count instead of the key. * fix(scim): lock member rows while recomputing admin roles after a group push * fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member * fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member * fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time. The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team. A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup. * fix(user-delete): keep the team alias label on the team members gauge after removals * test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them * fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404 * fix(scim): answer 404 and roll back when the team is deleted during a roster write * test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast * test(scim): cover deleting a SCIM group through the team delete's bulk path * fix(scim): evict a deleted group's member caches in bulk and drop cached roles after a recompute * fix(scim): answer 404 to a group delete that lost a race to another delete * fix(scim): chunk the locked team lookup and delete in the team delete transaction * fix(scim): keep service-account keys and team models on a group delete and write its records once under the lock * fix(scim): demote a deleted group's members inside the delete transaction The admin-group role write now runs on the team delete's transaction, over the roster read under the team lock, so a failed delete leaves roles untouched and a member added while the delete waited is demoted with the rest. The delete transaction also waits for a free connection as long as it may run, so a burst of deletes queues instead of failing after Prisma's two-second default. Every member in the 5-vs-500 statement test now holds a key. * fix(audit): write deleted keys' audit rows in one insert A delete that removes many keys wrote one audit insert task per key, and those tasks held pool connections the delete's follow-up writes waited on. The race test now waits on any lock wait behind the delete.	2026-10-21 00:59:23 +00:00
Kerry Lu	27efc387f7	fix(proxy): price earlier timed-out retry attempts at their own deployment's model rate (#47648) ... * fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate * fix(router): read the retry record's base_model through the typed metadata helper * fix(router): keep the four-record retry breadcrumb cap * fix(router): carry a deployment's litellm_params base_model on its retry record * chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 22:26:09 +00:00
Mateo Wang	c4360bbb59	fix(harness): set PWD to the working directory of local sandbox commands (#47530) ... * fix(harness): set PWD to the working directory of local sandbox commands LocalSandbox started CLIs in the sandbox workdir but passed the parent's PWD through. OpenCode reads PWD as its project directory, so a litellm.agent() turn worked in the Python process cwd instead of the sandbox. * fix(harness): re-home resumed OpenCode sessions into the sandbox folder OpenCode files each session under the folder it ran in and only streams a resumed session's events from that folder, so a session saved before PWD pointed at the sandbox, or resumed in another sandbox, hung. Resume now runs opencode export and import in the sandbox first. Also types the CLI handler's config so its member accesses are known. * fix(harness): fail an OpenCode resume that cannot move its session, with a timeout The export and import step now has a 30 s bound, and a failure or timeout raises HarnessError instead of leaving the resumed turn waiting forever. A restart of a session the handler already ran skips the step. * fix(harness): resume an OpenCode session elsewhere as a copy and fail the turn when it cannot Importing the exported session moved it, so the original stopped working where it was saved. Resume now imports a copy under fresh ids, filed under the sandbox folder, and continues the copy. A failed or timed-out export or import ends the next turn with runtime_error, as base does for a missing session, instead of raising from agent_resume. * fix(harness): retry a failed OpenCode resume on the next turn and keep resumed copies stable Resume now only records the saved id, and the first turn runs the OpenCode export and import under the caller's timeout. A failed export fails that turn and the next turn tries again. The local sandbox kills a command when its run is cancelled, and the copy names the sandbox folder so OpenCode 1.15.5 and older keep it there. * test(harness): keep subprocess tests out of tests/unit The PWD regression test moves to tests/integration/sdk with an isolated child interpreter, and the cancel test checks collect_output with an in-process process instead of a real one. collect_output takes the sandbox Process protocol. * fix(harness): rewrite only id fields when copying an OpenCode session for resume	2026-10-20 20:06:11 +00:00
Mateo Wang	c20d57c18a	fix(ui): rebuild the dashboard bundle and cover it in the example master key check (#47612) ... * chore: update Next.js build artifacts (2026-10-20 18:15 UTC, node v24.19.0) * test(security): cover the generated dashboard bundle in the example master key check * test(security): keep a clean-repo success case for the example master key check	2026-10-20 13:02:29 -07:00
Fábio Antunes	0ca29c7d84	feat(harness): add pi harness for litellm.agent() (#45467) ... * feat(harness): add pi harness for litellm.agent() Adds Harness.PI, which drives the pi coding agent CLI (`pi --mode json`) once per turn against the per-session model endpoint. The endpoint is declared as a `litellm` openai-completions provider in the models.json of a LiteLLM-owned PI_CODING_AGENT_DIR, so the user's own pi settings, auth, extensions and MCP servers are never read, and the token travels only through an env var referenced as `$LITELLM_HARNESS_TOKEN` pi exits 0 after a failed provider call, so a turn fails when the last assistant message ends with stopReason error/aborted. Permission modes map to a `--tools` allowlist, disable_tools to `--exclude-tools`, and sessions resume with `--session` from a persisted session dir. build_instructions and turn_prompt move from the OpenCode config into the shared harness utils so both configs use them Unit tests run on JSONL recorded from pi 1.1.0 * test(harness): align pi harness tests with the other CLI harness suites Mirror the OpenCode suite's layout, fakes and helpers, and add the cases every CLI harness covers: turn before start, early close kills the process, persist failure, endpoint model fallback, long JSONL lines, skills without SKILL.md and a missing endpoint. Add an endpoint_requests.jsonl fixture recording what pi sends to the session endpoint, and re-record the turn fixtures in one run so their ids agree * fix(harness): parametrize harness e2e only over harnesses with a model group harness_params() iterated every Harness member, so Harness.TOOL_LOOP, which has no entry in BINARIES or DEFAULT_MODEL_GROUPS, raised KeyError and the whole suite failed to collect * feat(harness): accept pi config and MCP servers through PiOptions.config PiOptions gains `config`, like the other harness options. `mcpServers` is written to the private agent dir's mcp.json and every other key to its settings.json. Keys that would reroute model calls, bypass permissions=, or load code as the host user (defaultProvider, defaultModel, defaultTools, defaultProjectTrust, extensions, packages, ...) raise OptionsMismatch A plain-name `--tools` allowlist drops MCP tools in pi 1.1.0, so when servers are configured the allowlist also gets `mcp__*` plus the tool each exposure is reached through (`codemode` by default, `tool_search` for `deferred`). disable_tools still excludes individual MCP tools * refactor(harness): give harness options immutable mapping defaults `field(default_factory=dict)` left every config/env default typed as `dict[Unknown, Unknown]`, which basedpyright reports as reportUnknownVariableType under strict mode: 9 errors across the five options classes, two of them added by PiOptions. A shared module-level `MappingProxyType({})` default is accepted by frozen dataclasses, matches the no-mutation rule the rest of the harness follows, and drops the file to zero errors so the rule shrinks against the merge base instead of growing. * fix(harness): reject managed env keys and a missing model in the pi harness PiOptions.env could name PI_CODING_AGENT_DIR or LITELLM_HARNESS_TOKEN and the value was dropped in silence, because transform_session_setup merges the managed keys last. The ordering stays as defense in depth, but validate_environment now rejects those keys up front the way the Claude Code config does, so a bad option is an error rather than a setting that appears to apply and does not. transform_turn_request also stringified the model it resolved, so a call with neither model= nor an endpoint model would have put the literal "None" on pi's --model. Both call sites now go through session_model(), which raises the same ValueError transform_session_setup already raised. * test(harness): escape the regex dot in the pi skill manifest match * fix(harness): keep harness options importable on Python 3.11 Python 3.11 rejects an unhashable dataclass default, and MappingProxyType only became hashable in 3.12, so the shared MappingProxyType({}) defaults made importing litellm.harness.options raise ValueError and broke every harness on 3.11. The defaults now come from typed default factories that still return read-only mappings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(harness): address review feedback on the pi harness tests Drop the endpoint-request fixture test, which only read a recorded file and could not fail when PiHarnessConfig changed, along with its now unused fixture. Remove the section-divider comments and a comment that restated MANAGED_ENV_KEYS, and type the test helpers and fakes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(harness): cover the last uncovered pi harness lines Assert that a tool_execution_end event without a result yields an empty ToolResult output, and that a PiOptions.config value JSON cannot encode raises TypeError. The Python 3.11 default check now asserts on the list of unhashable defaults instead of relying on hash() raising, which the test-quality gate flagged as an assertion-free test * test(harness): annotate harness option tests and stop rebinding helper params Mark the option test locals Final and check read-only defaults through the MutableMapping interface instead of writing into them. In the pi suite, type the parametrized test arguments, and have started() and parse_all() bind new names instead of reassigning their sandbox and state parameters * test(harness): test harness option defaults by constructing the options Replace the field-introspection check with one that builds each options class twice and asserts every mapping default is empty, read-only and not shared between instances. Sharing one MappingProxyType across instances is what broke class creation on Python 3.11, and that sharing is observable on every Python version, so the test fails on the old code without inspecting dataclass declarations * test(harness): stop mutating test helper inputs and assert whole values make_ctx took model and endpoint out of **kwargs with pop, which mutated its own parameter; they are explicit parameters now, with a frozen default endpoint. assistant_end builds its message in one expression instead of adding errorMessage afterwards. Two docstrings that explained an assertion became that assertion's message, and the options test compares whole values so a failure names the offending fields * ci(unit): run the pi harness unit tests in the llms-h-to-z shard main now lists every tests/unit/llms provider directory per shard, so the new tests/unit/llms/pi directory was claimed by no shard and assert_ci_coverage failed on both the shard and the test-file checks * test(harness): mark pi test locals Final and freeze the test output model Every single-assignment local and module constant in the pi suite is now Final. The name-mapping loop became one comprehension compared as a whole list, test_start_persists_sessions_dir takes the sandbox by index instead of binding _ twice, and the Answer output model is frozen like every other pydantic model * fix(harness): send PiOptions.thinking to the gateway as reasoning_effort pi clamps --thinking to "off" for a model that does not declare reasoning, and the generated models.json only declared {"id": model}, so every thinking level was dropped and the gateway never saw reasoning_effort. xhigh and max are also only offered when thinkingLevelMap names them. The model now declares reasoning plus that map whenever a level other than "off" is requested, and nothing when thinking is unset, so pi does not send its default effort on every call The pi tests also drop **kwargs from make_ctx and started in favour of explicit parameters, and annotate their helpers and the fake sandbox with read-only types; the fake records calls by rebinding tuples instead of appending to lists * docs(harness): list pi among the harnesses in the module and agent() docstrings * fix(harness): keep pi calls provider-agnostic and harness option defaults copyable Declare supportsStore false so pi stops sending store, take contextWindow and maxTokens from the model map, treat a length stop or a silent exit as an error, and give every options class plain dict defaults again so deepcopy, pickle and asdict work * fix(harness): look up pi gateway groups as litellm_proxy and ask pi for cache_control marks A gateway session's model is the bare group name, so the model map lookup now passes custom_llm_provider=litellm_proxy, the way the SDK prices it. That keeps an unmapped group from printing get_llm_provider's provider list banner into the caller's stdout. pi only adds Anthropic cache_control marks for OpenRouter Anthropic models, so every Claude call through the harness missed the prompt cache. Models the map says support prompt caching now declare cacheControlFormat anthropic, which LiteLLM translates for each provider. * fix(harness): cap pi's maxTokens at its own default and fork sessions saved in another directory pi sends maxTokens as max_completion_tokens on every call, and the gateway reserves that many tokens against the key's TPM limit, so passing the model's full output cap (128000 for claude-haiku-5-5) made keys with lower limits answer 429. maxTokens is now the lower of the model's cap and pi's own 16384 default. agent_resume() into another sandbox directory stopped at pi's interactive fork prompt. The native session id now carries the directory pi saved the session in, and a resume from a different directory passes --fork, so the conversation continues in the new sandbox. * fix(harness): leave pi room for the thinking budget and price cached prompt tokens pi's maxTokens now adds LiteLLM's thinking budget for the requested level to pi's 16384 default, within the model's output cap, so Anthropic budget thinking at max no longer answers 400. Harness turns now pass cache read and write tokens to cost_per_token, so cached prompts are priced at their cache rates on every harness. * test(harness): cover cached prompt cost on the gateway and non-stream routes * test(harness): mark the cost test constants Final * test(harness): type the cached cost test helpers without Any * fix(harness): price OpenAI cache write tokens in harness cost --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 09:01:40 +00:00
Mateo Wang	d698106ed7	fix(router)!: schedule default_priority requests once and reject a malformed priority (#47498) ... * fix(router): schedule default_priority requests once and reject a non-integer priority A router configured with default_priority failed every request that carried no priority with a 500 naming a missing 'priority' argument, and an explicit priority re-entered acompletion and hit the same error. acompletion now resolves the priority once (the request's integer, else the default, with 0 kept as a real priority) and schedules through _schedule_factory directly, while schedule_acompletion delegates to acompletion so the two paths share one queue pass. A non-integer priority answers 400 naming the param, or is dropped under drop_params, instead of leaking to the provider or crashing the scheduler. * fix(router): honor router-level drop_params and schedule prompt-management models under a priority * fix(router): drop a non-integer priority when the model's deployments set drop_params * fix(router): resolve the drop_params deployment scope for team routes and deployment ids * fix(router): read the request and router drop_params flags without a loop-assigned Final * refactor(router): type the prompt-management wrapper's return and mock the provider in its test * perf(router): consult drop_params for the priority only when it is not an integer * test(integration): assert the new priority contract for non-integer and zero priorities * docs(router): describe default_priority as the priority every request without one takes * fix(router): parse default_priority once and apply it to text completions and the queue route A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig. * fix(router): type the priority dispatch so the lint budgets hold * fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order * fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses * fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority A request whose model is a deployment id now counts that deployment as healthy in the scheduler, since routing by id already ignores cooldown. Before, the scheduler found no healthy deployment for an id, so the request waited behind every entry ahead of it. A boolean default_priority (true, or YAML off) is ignored at startup with a warning, like any other value that is not a whole number. The scheduler integration tests now share worker state for the defaulted pair, cool a group through /v1/responses so no stale in-memory queue is left behind, and use "high" instead of "1" for the non-integer cases. The concurrent burst test no longer asserts that the queue drains, matching the explicit-priority burst test, since the scheduler's read-modify-write queue can race across workers. * test(router): keep the queue drain check as a BUG-skipped integration test * fix(router): queue default_priority chat requests once instead of failing them * test(router): open with the default_priority regression tests alone * fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name	2026-10-20 08:00:57 +00:00
Mateo Wang	8a2782e41c	feat(router): route batch creates to mode batch deployments and expose supported_endpoints on /model_group/info (#47376) ... * feat(router): expose supported_endpoints on /model_group/info * feat(router): route batch creates to mode batch deployments in mixed model groups * fix(router): prefer batch deployments after eligibility filters and in group-name credential lookups * fix(router): keep group-name credential lookups on the first deployment and prefer batch deployments before cooldown Model-encoded ids carry only the group name, so preferring the batch deployment there moved ids created before the upgrade onto its credentials. Batch creates now pick batch-capable deployments before health checks and cooldown, routing and group info share one batch-capable check, the cost map's /v1/batch is dropped only for Azure, and /model/info applies the same per-deployment rule * test(router): map _listed_endpoints to its /model/info tests in the router coverage check * fix(router): skip only Azure chat deployments for batch creates and list batch support per caller team Narrow the batch create filter to Azure and azure_ai deployments that are not mode batch, so OpenAI groups keep their chat deployments during cooldown and under tag filters. Mark batch deployments by mode batch only, since stored endpoint lists from older Admin UI saves copy the cost map. Leave /model/info unchanged so the Admin UI never saves a computed endpoint list, and pass the caller's team into /model_group/info. * test(proxy): let the model hub router fakes take the caller's team_id * fix(router): gate batch-create routing on deployments the caller's team may use and list only /v1/batch for Azure batch deployments * fix(router): skip a malformed model_info.supported_endpoints instead of failing group info and chat * fix(router): apply the batch create filter after cooldown, managed files, and tags The batch filter ran before health checks and cooldown, so a batch create on a mixed Azure group answered with a router 429 or 401 when cooldown, tags, or a managed file's targets left no batch deployment, where main sends the create on and returns Azure's 400. The filter now runs right before order filtering, after every eligibility filter, so those cases answer as main does and creates still skip Azure chat deployments whenever a batch deployment is left * test(router): annotate the batch routing test assignments with Final	2026-10-20 07:01:12 +00:00
ekpog200	ba61ecdf57	fix(videos)!: keep model_id in proxy video ids so per-model keys resolve on status/content (#44614) ... * fix(videos): keep model_id in proxy video ids so per-model keys resolve on status/content Provider transforms encode returned video ids with an empty model_id, so GET /v1/videos/{id} and /content cannot resolve the deployment (and its per-model api_key) and fail with invalid_api_key. Re-stamp the returned VideoObject id with the resolved model_id in video_generation, video_status, video_remix, video_edit and video_extension. Fixes #33423 * fix(videos): return a copy instead of mutating the response, read hidden params via helper * test(videos): fully type the new video id re-stamping tests Annotate every parameter and return added by this PR (hidden_params, harness, response, -> None), drop Any from the test helper, mark locals Final, and narrow the endpoint/helper results with isinstance(VideoObject) instead of reading .id off an untyped value. Route the mocked processor response through a typed Harness.respond_with, read NotRequired DecodedVideoId keys with .get, and name the rule and reason on the _hidden_params suppressions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(videos): fill only an empty model in returned video ids Keep an id that already names a model as is, so the asked-for name, routing group and served group in it still drive access checks and routing. Fill an empty model from the model the caller asked for before the routed deployment id, and read the request model through a typed view so the type gate does not grow * fix(videos): route video calls to the deployment that created the video Video ids now carry the id of the deployment that created them. Status, content, remix, edit and extension calls go to that deployment when it belongs to the requested group, so a group spread over two provider accounts no longer polls the wrong one. A provider model shared by several groups resolves to the creating deployment's group, and the asked-for model is written into an id only when the router can route it. * fix(videos): pin video calls only to deployments the caller may use Pin a video id's creating deployment at deployment selection, and only when the caller may use that deployment in the requested group, so aliases, routing groups and access groups still apply and the spend label and fallbacks keep the requested group. A status poll keeps the deployment its id names and takes the deployment that answered when the id names none. An alias that routes to the creating deployment's group is kept on status and content. * test(router): reset module fallback globals between unit tests * fix(videos): check and route polls by the name the router serves A poll on a video created through a provider wildcard is checked against the name the wildcard serves, a team model is checked by its public name, a poll stamps the deployment that answered, and the sync Router pins video calls the same way the async path does * fix(videos): pin polls only among the deployments routing allows Video calls route by group again and the creating deployment is picked only from the deployments left after access, team, health, cooldown, budget and tag filters, so a pinned poll no longer skips them. Wildcard groups now pin too. Adds tests for cooldown, wildcard pinning and character lookups through a provider wildcard, and return types on the auth tests * fix(videos): scope the video pin to video calls, keep fallback-served videos reachable, skip cooldown on video 404s * fix(videos): stamp the requested group only when one of its router fallbacks served the create * fix(videos): follow chained router fallbacks for fallback-served videos, test with a real Router * test(router): list the video pin helper with its sibling in the router coverage exemptions * refactor(auth): walk chained router fallbacks without recursion --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 05:42:59 +00:00
Mateo Wang	e34c779c67	fix(proxy): decode zstd and gzip request bodies before auth reads them (#47418) ... * test(proxy): cover gzip and zstd request bodies on /v1/systemone * fix(proxy): decode zstd and gzip request bodies before auth reads them * fix(proxy): bound request body decompression and keep it inside CORS Limit the decoded size by an expansion ratio and the operator's max_request_size_mb, pass bodies without the coding's magic number through, answer 415 when zstd is unavailable, and register the middleware innermost so errors carry CORS and security headers * fix(proxy): budget decoded request bodies per worker and forward plugin bodies as sent Each worker holds at most MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB of decoded bodies and answers 503 past it. /plugin-proxy/ bodies reach the plugin as sent, a zstd skippable frame counts as zstd, the decoded-size 413 names the decoded limit, and an integer max_request_size_mb stays an integer in the 413 text. * fix(proxy): read the magic number before size limits and reserve declared decoded sizes * test(proxy): pin gzip mtime so parametrized test ids match across xdist workers * fix(proxy): parse max_request_size_mb the same way in the auth size check * fix(proxy): pass Claude Code OTLP export bodies through undecoded and skip the size parse without premium * test(proxy): cover client disconnects, chunked bodies past the cap, and body-less request messages in the decompression middleware * test(proxy): name the decompression test cases so CI logs stay small	2026-10-20 05:11:30 +00:00
devin-ai-integration[bot]	6fc60bc00d	feat(router): discover kubernetes pods behind a headless service api_base (#44342) ... Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 09:58:19 -07:00
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
yuneng-jiang	44fcbc30b0	test(mcp): replace the live Linear MCP OAuth e2e tests with integration coverage (#47154) ... * test(mcp): move the MCP OAuth happy path from a live Linear e2e to an integration test GitHub Actions now runs unit tests only, and the MCP OAuth happy path needed a captured Linear browser login to click through a real consent page. The OAuth flows it exercised are already integration tests against an owned authorization-server double, except one behavior: the stored per-user token must be served by a proxy process that never saw the consent, with no shared Redis to fall back on. That case is now in test_mcp_oauth_flows.py The e2e test, its owned-gateway helper, its workflow, the mcp_oauth_live marker and the JUnit gate script only that workflow used are removed * test(mcp): chat completions run an OAuth server tool with the caller's own stored token * test(e2e): drop the Linear chat-completion MCP OAuth e2e and its helpers The behavior it checked, a chat completion running an OAuth server's tool with the caller's stored per-user token, is now an integration test in test_mcp_llm_endpoints.py that needs no saved Linear browser login * test(mcp): decode the stored per-user credential as urlsafe base64 The plaintext check used b64decode, which drops the - and _ of a urlsafe value and raised on roughly one ciphertext in five * test(mcp): drop the chat OAuth integration test that duplicates test_mcp_slots F2-chat security/test_mcp_slots.py already sends a per-user OAuth token through /v1/chat/completions and asserts the peer receives it. Every mutant the new test killed was also killed by the existing suite, so it added no signal	2026-10-17 14:29:59 -07:00
ryan-crabbe-berri	f6ae25939e	feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) ... * feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries * fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5 A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow * refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass * chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py	2026-10-17 12:22:41 -07:00
Mateo Wang	d76ec3a47e	fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583) ... * fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing A non-list tools value crashed the vector store lookup during auth and the built-in tool cost lookup in the failure logging path, so the proxy answered an unhandled 500 and closed the keep-alive connection. count_tokens now answers a 400 invalid_request_error for a malformed body, drops malformed tools when drop_params is on, and both lookups skip values of the wrong shape. * fix(proxy): skip non-list tools in the tool allowlist check Also validate the count_tokens body without recursion, which the recursion detector rejects. * test(integration): restore the int tools case on count_tokens The proxy now answers a non-list tools value with a 400, so the case is deterministic again. * fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids * fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids Check tools after request setup so the Claude Code default and the logged call type apply, read the request, global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off * fix(proxy): keep the request's call type when logging a gate rejection after request setup The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses rejection as aembedding, so it now only applies when the logging object carries no known call type * fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry * fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket * refactor(proxy): leave the responses websocket out of the request-level tool check * fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections * fix(proxy): read router default drop lists and deployment ids in the malformed tools check * fix(proxy): leave malformed tools to the router when it can pick another deployment Fallbacks, auto-routers, model strings, comma batches and team model names let the router pick a deployment after the proxy checks the request, so the proxy now forwards a malformed tools field there as main does, and only rejects it when the deployment it checks is the one that serves. Token counting never reroutes, so count_tokens and input_tokens keep reading the group they count with. additional_drop_params names are read the way core reads them, so a list with a non-string entry still names its strings. * test(integration): cover malformed tools on reroutable models, router defaults and deployment ids * fix(proxy): resolve malformed tools at the deployment the router picks * fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells. * test(integration): expect a 400 for an empty-string tools field on /v1/messages Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test * fix(router): keep web search routing for tools passed as a tuple * fix(router): keep a deployment's own tools when drop_params drops a malformed request tools * test(router): list _merged_tools as covered through the deployment tools merge tests * fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them * fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError * fix(prometheus): keep deployment health metrics clean on a malformed tools rejection * test: type the new malformed tools tests * test: mark new test locals Final and type kwargs dicts	2026-10-16 14:57:53 -07:00
Ayush Shekhar	7930c0e434	feat(router): mid-stream fallback continuation for chat completions (#41127) ... * feat(router): continue chat streams on fallback after a mid-stream break A chat-completions stream that breaks after content has been delivered was re-raised and the fallback deployment never ran (#40404), while the Responses-API path already continues via a prefilled assistant turn. Add the same for chat completions behind `enable_mid_stream_fallback_continuation` (opt-in, async-only), so a post-content break re-enters the fallback chain with the partial text as an assistant prefill instead of surfacing the error. Continuation is only attempted when it is safe: the stream emitted plain assistant text (no tool/function calls, thinking blocks, reasoning items, audio or images), the request is not constrained output (response_format or a forced tool_choice) and not merge-reasoning mode, and the fallback target's model supports assistant prefill. The target check runs as a deployment pre-call filter, so a chain with no prefill-capable deployment empties and the original error is surfaced rather than a duplicated or rejected request being sent. Plain reasoning_content is treated as out-of-band and does not block a continuation, matching the Responses-API path; Anthropic thinking is excluded because its signed thinking blocks cannot ride a text-only prefill. CustomStreamWrapper now latches whether a disqualifying delta was streamed and carries it on MidStreamFallbackError so the router can decide without re-scanning chunks. Default behavior is unchanged. * fix(router): harden mid-stream continuation eligibility and prefill folding Address review of the mid-stream continuation: - keep response_format={"type": "text"} eligible; only json_object / json_schema and other structured formats decline, since "text" is the unconstrained default - fold a new partial into an existing trailing assistant prefill instead of appending a second one, so a nested mid-stream break stays a single prefill turn even on providers that do not merge consecutive assistant messages - split the deployment prefill-capability lookup into two readable steps - cover the merge_reasoning decline branch, the text response_format pass, and the prefill-folding path with tests * fix(router): address review of mid-stream continuation - register the continuation deployment filter as a process-global singleton instead of tracking it per router, so discarding one router no longer removes the type-deduplicated filter that other live routers still depend on - disqualify reasoning_content: it reaches the caller as visible reasoning that a text-only prefill cannot carry, so a reasoning-capable fallback would re-derive it and produce an incoherent response - move MID_STREAM_CONTINUATION_KWARG to constants and lazy-import the filter class in the router, breaking the module-level import cycle CodeQL flagged - trim the added docstrings to the repository comment policy * fix(router): keep mid-stream continuation within the type-discipline budget The LIT* type-discipline gate counts mutable-collection use the ruff pass does not see. Annotate the reads with read-only Sequence/Mapping views, compare response_format against a frozen MappingProxyType sentinel, and mark the genuinely mutable litellm message and deployment-list boundaries with mutable-ok reasons, so LIT001 and LIT002 stay under their limits. * fix(router): clear basedpyright operator error and the import-cycle alert - narrow the accumulated stream delta content to str before appending, so the shared helper no longer trips reportOperatorIssue (str += object) - keep the continuation marker in the pre-call-check module and lazy-import it at the two router call sites instead of routing it through constants, so the router has no module-level dependency that closes an import cycle * fix(router): make the mid-stream continuation marker unforgeable The deployment filter is process-global and read the continuation marker as a truthy value from request_kwargs. Since the proxy can forward arbitrary request-body fields into the router, a client could send `"_mid_stream_continuation": true` on a normal request to filter a mixed model group down to prefill-capable deployments and steer their prompt onto that provider. Set a private sentinel object internally and check it by type instead of truthiness: a JSON request body cannot construct one, so a forged flag is ignored. Adds a regression test that forged client values (true, "true", 1, a dict) leave the deployment list untouched. * feat(router): honor per-deployment model_info for continuation prefill capability The deployment filter only read supports_assistant_prefill from the cost map, so a model configured with `model_info: {supports_assistant_prefill: true}` on the deployment (the usual way to declare a capability for a model that is not in the cost map, or to override it) was wrongly skipped. Check the deployment's model_info first (True or False both authoritative), then fall back to the cost map. Adds tests for the override in both directions. * fix(router): bound mid-stream continuation to the caller's output allowance A continuation handed the fallback the full max_tokens / max_completion_tokens even though part of the answer had already streamed, so the total could exceed the caller's limit and repeated interruptions could re-grant the allowance on every hop, amplifying spend. Reduce each configured ceiling by the tokens already emitted before re-entry, and decline the continuation (surface the original error) when the allowance is exhausted. The reductions compound across hops because the trimmed ceiling is what the next hop sees. Also add the missing explanatory comment on the model_info fall-through except. * refactor(router): extract mid-stream decline to stay within the complexity budget The output-ceiling branch pushed stream_with_fallbacks past the C901 limit. Both decline paths (ineligible after content, and exhausted allowance) re-raised the original provider error the same way, so pull that into _raise_original_mid_stream_error. It removes the duplicated branching, drops the function back under the complexity ceiling, and reads clearer. * fix(cost-map): mark Claude Sonnet 4.6 as not supporting assistant prefill The Anthropic API and Bedrock both reject an assistant-last message on claude-sonnet-4-6 with "This model does not support assistant message prefill", so the continuation filter must not route a mid-stream continuation to it. Flip the flag on every claude-sonnet-4-6 entry and on databricks' opus-4-6 entry, whose anthropic sibling already says false. The prefill check tests now register synthetic cost-map entries instead of pinning real models, which broke when claude-3-opus left the map. * refactor(router): type the streaming fallback kwargs and drop narration comments Removes an unused mutable-ok marker (LIT013) and the explanatory comments Greptile flagged, and types initial_kwargs on both streaming fallback iterators as dict[str, Any] like their Responses siblings so the new continuation helpers take a typed mapping and the basedpyright budget stays at or below main. * fix(router): run post_call_rules over the joined text of a continued stream Each provider stream wrapper validates only the fragment it produced, so a rule that trips on the primary's prefix plus the continuation's text never fired. The router now feeds litellm.post_call_rules the emitted text plus every continuation chunk before yielding it. Also drops a narrating test-helper docstring. * refactor(router): read a deployment's declared prefill support without an empty except * test(router): cover _raise_original_mid_stream_error directly for the router coverage gate * fix(router): fold the continuation prefill into a trailing assistant turn A request that already ends in an assistant message took the partial output as a second assistant turn, which prefill providers reject. The partial now extends that turn in place, with prefix set, so the continuation request never ends in two assistant messages. * fix(router): keep content blocks on a trailing assistant turn when extending it with the partial output A trailing assistant message whose content is a list of blocks was stringified before the partial output was appended, so the fallback received the Python repr of the blocks as its prefill. The partial now lands as one more text block, and a content shape that cannot be extended surfaces the original stream error instead of a garbled continuation. * fix(router): continue the trailing text block in place when extending a prefill with content blocks Anthropic strips trailing whitespace per content block, so appending the partial output as a new text block after an assistant turn that ends in "Sure, " would send the prefill as "Sure,here". The continuation now extends the last text block itself and only opens a new text block when the trailing block is not text (an image, say), so the prefill reads exactly as the text the caller already received * test(router): type the prefill check's deployment helper * fix(router): keep a continued stream on the primary's identity and widen what can continue Continuation chunks now keep the primary stream's id and created time and drop the fallback's second role delta. Requests with n > 1 no longer continue. Prefill support reads model_info.base_model and litellm_params.base_model before the deployment model. enable_mid_stream_fallback_continuation is runtime-updatable through update_settings and registers the prefill filter. The bridge rebuild of MidStreamFallbackError keeps the inner stream's disqualifier. A fallback that repeats the prefill is not doubled in the continuation input. * fix(router): decline mid-stream continuation after a content filter, merged reasoning, or a plain trailing assistant turn Also count the partial stream's completion tokens in the continued stream's usage, accept string values for enable_mid_stream_fallback_continuation, and list the new router helpers in the router coverage check * fix(router): drop re-emitted whitespace at the continuation join and sum both legs' usage consistently * fix(router): keep the full prefill so a nested break keeps the separator * fix(router): bill both calls of a continued stream and fix Bedrock Converse, Databricks, and Vertex DeepSeek prefill routes * test(router): integration cells for mid-stream fallback continuation Covers the continued stream the caller gets, the prefill turn and reduced max_tokens the fallback receives, the prefill-capability filter, the runtime toggle, repeated requests through the response cache, and the spend row of a continued stream * fix(router): log a continued stream under the caller's id, price each cut on its own prompt, and keep prefill flags per deployment * fix(router): price and log each cut leg before its continuation reuses the logging object Price the cut stream's partial and let its failure log finish (bounded) before the fallback overwrites a shared proxy logging object, and skip continuation when an output guardrail is registered or a request fallback carries its own messages. * test(router): integration cells for the output-guardrail and request-fallback declines and the cut call's failure log Cover a registered post_call guardrail and a request fallback carrying its own messages keeping the original error, an input-only guardrail leaving continuation on, and the cut call's failure event naming its own deployment and price. * fix(router): skip mid-stream continuation when a request fallback overrides request params A request-body fallback entry that sets anything besides model (stream, messages, max_tokens) changes the request the prefill continuation relies on, so a stream: false entry ended the answer with no error. * fix(router): skip mid-stream continuation for a fallback group target that overrides request params The override check read only top-level fallback entries, so a target inside a group's list, such as {"primary": [{"model": "backup", "stream": false}]} in the request, the config, or a key's router settings, still ended the answer with no error. * fix(router): check only the request's own fallback chain for request param overrides Scanning every group's chain turned continuation off for all requests when one unrelated group's target overrode a param. Resolve the chain the router will use for this request, the same way the fallback path does. * fix(router): keep the original error when request fallbacks are malformed The request-param override check resolved a standard-shape chain outside the router's own error handling, so an empty entry or a non-list chain raised StopIteration or TypeError and replaced the original mid-stream error. Skip empty entries and treat a non-list chain as no chain. * fix(router): read deployment-configured constraints, decline deployment-id fallbacks without prefill, strip only the last Bedrock text block * fix(router): read a deployment's tool_choice ahead of router defaults in the continuation check, and restore the renamed prefill test names * fix(router): filter continuation deployments on every routing strategy, read extra_body and fallback deployments' own params The deployment filter now also runs on the sync selection path that usage-based-routing v1 and custom strategies use, one predicate reads n, response_format, tool_choice and merge_reasoning_content_in_choices with extra_body laid over them, fallback deployments whose own config constrains the output are skipped, and continuation declines when the cut deployment's id is not in the router (per-request credential deployments) * fix(router): decline a cut mid-echo, reduce extra_body output limits, and check post_call_rules once on nested continuations --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 13:51:32 +00:00
武本龍門	43be1c2ae7	fix(router)!: keep RPM, region, and param checks when token counting fails (#44544) ... * fix(router): stop skipping rate-limit checks when token counting fails Router._pre_call_checks() caught any token_counter() exception and returned the full, unfiltered deployment list, bypassing the RPM, region, and invalid-params checks for every deployment - not just the context-window check that actually depends on the count. A content block token_counter() cannot parse (e.g. an invalid image_url detail value) silently disabled rate-limit enforcement for that request, with only a log line and no error surfaced to the caller. Now a counting failure only disables the context-window check; the other checks still run for every deployment as before. Input-token resolution is pulled into its own method and computed once before the loop, so nothing needs to track "did counting fail" with a mutable flag across iterations. * test(router): cover the async region check when token counting fails Replace the unit test of the private input-token resolver with a regression test through async_get_healthy_deployments, the path the proxy uses: an EU-pinned request whose content cannot be token-counted must not get the US deployment * test(router): cover the param filter when token counting fails and satisfy router coverage * fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params, drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away instead of a retried 429. Deployment context windows are resolved once per call. * fix(router): keep the retried 429 when a cooling-down deployment takes the param * fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks * fix(router): steer by deployment capability, apply drop settings after region and RPM checks A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing * test(router): make the github_copilot pre-call filter test hermetic in CI * test(router): cover pre-call region and response_format filters through the proxy * fix(router): judge the 400 against the request's own deployments, not a name lookup * fix(router): judge the 400 against the passed deployments when the group name lists none * fix(router): let a request's drop and allow settings replace the deployment's in the param check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 09:20:34 +00:00
Michelangelo Setaro	c288dc7da0	fix(router): let an exact base_model widen a deployment's context window (#46197) ... * fix(router): honor explicit bedrock base model capacity * fix(router): honor an exact base_model's capacity for every non-Azure provider Extends the Bedrock-only check to every non-Azure provider and only uses the base_model when it has its own exact cost-map entry, so an unmapped or wildcard base_model falls back to the route id instead of raising in the router. * test(router): call _maps_to_exact_model_info directly in its cost-map tests * fix(router): let an exact base_model only widen a deployment's context window A base_model used as a pricing stand-in with a smaller window no longer narrows the route's limit, and a route with no window never takes one from base_model * test(router): count the base_model window helpers as covered through get_router_model_info * fix(router): widen only from a model_info base_model naming the same model A stand-in base_model for a different, larger model widened a smaller deployment's window, so pre-call checks sent it prompts the provider rejects with 400. The window now widens only when model_info.base_model names the same provider model id as the route, which is also the field /model/info reads. * fix(router): match a base_model's model id across Bedrock region prefixes and @ qualifiers * fix(router): read litellm_params.base_model too and look the base row up as written * test(router): cover a same-model base_model window on chat, messages, responses, and complexity escalation --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 06:37:40 +00:00
devin-ai-integration[bot]	e344dbd8db	feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) ... * feat(proxy): split Azure PTU deployments across teams by share A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on, routing hides the deployment from teams holding no share, the v3 limiter enforces each share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team Usage page report PTU-hours next to tokens. * fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name The PTU flag is read from the process environment on every check instead of through the secret-manager path, so a proxy with a hosted secret manager does not pay a round trip per request. A team's ceiling now carries the model's cached-input ratio and settlement charges cache reads at that ratio, matching how Azure sizes a PTU. A deployment registered for one team is found by its team_public_model_name for PTU-hours and the ceiling, since its model_name is rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and parsed_ptu_shares refuses booleans, strings, and non-string team ids, a fractional ptu_count reserves nothing, and models_processed counts distinct deployments rather than holdings. * fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations * fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id * fix(ptu): refuse ptu_shares naming a team that does not exist * refactor(ptu): keep Azure detection under llms/azure and check share teams with one query * fix(proxy): drop the reverted TeamIdSearchFilter import after merging main Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally * fix(proxy): chunk the ptu_shares team lookup under the IN-list bound * fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling * fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling * fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour * fix(ui): seed total_ptu_hours in the empty daily activity metadata * fix(ptu): size usage rows by the name the request used * fix(ptu): size a team's usage rows by the deployment it is served from * fix(ptu): never size a team's usage row by another team's reservation * fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length * fix(ptu): read a share map only through the terms registration requires A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and ptu_effective_from, or one that does not add up to ptu_count, is refused at registration but was enforced once loaded: every other team got the 400 and the share holder a ceiling while no flat cost accrued. _deployment_shares now reads the shares through ptu_terms, so such a row reserves nothing, the same as before the map was read. * fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off 6fb24d1d read ptu_shares through the pricing terms so a termless map reserved nothing, but with the flag off the router skips ptu_config_error for config.yaml entries, so such an entry loaded and served every team, against the rule that a declared split restricts access whether or not cost attribution is on. _deployment_shares reads the declared map again, and the router now raises ptu_config_error for a config-sourced entry declaring ptu_shares regardless of the flag, so a malformed split is refused at boot instead of being honoured or ignored silently. * fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling A name served by a wildcard route now resolves to the shared deployment behind it, so the limiter's ceiling and the team daily activity sizing both find it. Pass-through responses settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a reservation taken at admission is settled even when the success metadata lost the team. * fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids. * test(router): pin the wildcard route precedence that PTU sizing and the router share * fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was refused with a 422 and dropped at load. Fractional, infinite, boolean and string shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix, zero-token Responses usage, fractional-count and sizing-warning paths. * fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move * fix(proxy): write each PTU deployment day's team charges in one transaction The backfill counts a deployment day as priced once any team's row exists for it, so a rollup that wrote team A's share and lost team C's left team C unbilled for good. Each deployment day's team rows now go through one Prisma batch with the retry around the whole batch, so a day is fully written or not written at all and the next backfill prices it for every team. * fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes * test(router): list the cached PTU ceiling helpers in the router coverage check * fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force * fix(proxy): bill a rerun PTU day to the current share holders alone * fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list * fix(proxy): read the PTU flag from the secret manager first, like get_secret * fix(router): scope retry health lookups to the caller only for PTU share holders * fix(router): move the share-holder retry check into ptu_shares Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them * test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 18:44:30 +00:00
Mateo Wang	622daf3cdf	fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) ... * fix(scim): write a group's roster in one locked transaction instead of one member at a time POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the membership rows and the users' team tags in one transaction under the team's advisory lock, so a 500-member push runs a fixed number of statements instead of thousands. PUT now answers with the roster it just wrote * test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers * fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock * fix(scim): evict the cached team on a group rename and read a no-op roster from the writer A rename-only PUT or PATCH wrote the new alias and left the team cached under its id and old alias on every worker, so keys kept logging spend against the old team alias until the TTL expired. The group write now evicts the cached team (and broadcasts the eviction) right after the alias or metadata update. sync_team_roster reads the team from the writer when the plan names nobody, so a replica cannot hand back the pre-rename row; a plan already matching the roster returns inside the lock without writing, evicting, or emitting the members metric; the response is built from the written row, so lastModified reflects the roster write; and detaching a removed member bumps the user's updated_at like every other user write. * fix(scim): bump detached members' updated_at in UTC The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts into Prisma's naive timestamp column in the session timezone, so on a database whose session is not UTC the removed member's row landed hours away from the UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and have the bulk roster cell assert the bumped row sits between two reads of the database's own UTC clock taken around the replace. * fix(scim): chunk the group write's member lookups under the bind cap Every Prisma "in" filter the group write issues bound one parameter per member with nothing bounding the list, which check_unbounded_in_lists reports as 11 new findings. The single-field lookups now go through litellm.repositories.chunked_in (caller_transaction inside the roster transaction, per_chunk_ok for the role recompute), and the one cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two slices, and a member named only in the last slice still resolves. * fix(proxy): send bulk cache evictions in slices so the broadcaster drops none Veria pointed out that the invalidation publisher caps its pending tasks at 1,024 and drops the excess with a warning, and that a group push now detaches every removed member's keys in one call instead of one member at a time, so a removal of more than roughly 340 members (three cache keys each) silently left stale key objects on the other replicas. The merge base never hit the cap because each member was a separate request. Bulk evictions (team, user, organization, and session key deletes, plus the roster settle) now publish in slices of 256 and wait, bounded at two seconds, for the backlog already handed to the event loop before each slice. A Redis that never answers costs a bulk caller at most that wait per slice instead of wedging the request, and single-key callers are untouched. Regression tests push 1,500 keys and a 500-member removal through a slow publisher and assert every eviction reaches Redis; both fail against the previous tip. * fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts Greptile pointed out that pacing the broadcasts in slices also paced the local eviction: a bulk delete waited for the publish backlog before dropping the next slice from this worker's own memory, so on a Redis that accepts connections but never answers a key deleted from the database kept authenticating on the handling worker for up to two seconds per slice. The merge base cleared every key locally at once. Bulk evictions now drop every affected entry from this worker's memory first, through the cache's own partition lookup since key objects live apart from the other auth objects, and only then publish in paced slices. The roster settle does the same for the deleted tokens, the JWT mapping keys and every changed member's user and membership entries before any step that can wait. Regression tests run both paths against a publisher that never answers and check that nothing is still cached on this worker before the first bound elapses; both fail with the local eviction removed. * fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts * docs(proxy): drop the module docstring that repeated the PR description * fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps Bulk key evictions now drop entries from memory, delete them from Redis in one chunked DEL, and broadcast every key in one pipelined publish, so a large roster removal no longer overflows the publisher's pending cap or costs a Redis round trip per member. A push waits for in-flight membership loads before evicting. User rows are locked in byte order to match the spend writer, a bulk user delete takes every team lock before its first user write, and a POST whose roster write fails deletes the empty team so the IdP retry creates it again. * test(team): observe delete broadcasts at the Redis layer * fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently * fix(team): chunk the email lookup behind the member row lock * fix(scim): create a member deleted mid-push again instead of failing the group write * fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch A member deleted mid-push is created again only when scim_upsert_user allows it, under the verb's own created_via. New group members are checked against the license seat limit as one batch before any is created. Auth cache publish drops log the key count instead of the key. * fix(scim): lock member rows while recomputing admin roles after a group push * fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member * fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member * fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time. The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team. A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup. * fix(user-delete): keep the team alias label on the team members gauge after removals * test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them * fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404 * fix(scim): answer 404 and roll back when the team is deleted during a roster write * test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast	2026-10-15 10:57:00 -07:00
TheMentalCoding	ca04c4909e	feat(ui): filter active and paused model deployments (#43710) ... * feat(ui): status filter + sort and provider-aware search on Models & Endpoints The models table mixes active and paused deployments with no way to tell them apart beyond the pause toggle, and the drawer only filters by public model name and access group. Backend (/v2/model/info): - new optional `blocked` query param to filter by routing status - new `blocked` sort field (asc = active first); the existing `status` field keeps sorting by config-vs-DB source - `search` now also matches litellm_params.model, so typing a provider or upstream model id (e.g. "nvidia_nim", "openai/gpt-4") finds deployments whose public name doesn't mention them Frontend (Models & Endpoints): - visible Status column with Active/Paused badges, sortable (active first on asc) - Status filter (All/Active/Paused) in the Filters drawer, wired to the new server param through URL state The routing-status filter ignores non-bool sentinels the same way exclude_auto_routers does, so direct calls bypassing FastAPI keep their no-filter behavior. * fix(ui): inverted routing-status filter mapping and compact Status column Selecting "Active" in the Filters drawer sent blocked=true to the server, showing the paused deployments instead of the active ones (and vice versa). The mapping now lives in routingStatusToBlocked() with unit tests, and the Status column is narrower so the row actions stay in view. * fix(ui): pin the Actions column to the right on the models table With the new Status column the table overflows on narrower screens and the pause/resume switches (last column) scroll out of view. Pinning the Actions column keeps them visible; uses the existing DataTable meta.pinned support (same pattern as MemberTable). * refactor: readable db search where-condition and ruff-format clean * fix: apply routing-status filter inside the search flow and regen API types Greptile flagged (P1): with search + a status filter combined, the bounded DB fetch ran before status filtering, so matches of the other status could consume the page budget and leave short pages / stale totals. - extract _matches_routing_status() (sentinel-safe) and apply it to the router-side matches inside _apply_search_filter_to_models - push the status condition into the DB where clause of _fetch_db_models_for_search, so rows of the other status never count against the bounded fetch - endpoint passes blocked through to the search flow; tests cover the combined search+status case and the generated where clause - regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts for the new query param - update useModels.test.ts expectations for the new trailing argument * fix(proxy): keep the flat DB where shape for single-condition searches The upstream CI test test_apply_search_filter_honours_exact_model_name_in_db_query pins the Prisma where shape: exact model_name stays a top-level key, and plain substring search keeps {model_name: {contains, mode}} at the top level. Only genuinely multi-condition queries (status filter and/or router-id exclusion) now wrap in AND. * fix(proxy): avoid reassigning a Final variable in the db where builder basedpyright's reportGeneralTypeIssues budget counts the reassignment of the Final-annotated db_where_condition; express it as a single conditional expression instead. * chore(deps): bump pypdf to 6.19.0 to clear osv-scan pypdf 6.16.2 has 8 High CVEs (GHSA-5jq2-8x83-x246 and aliases) fixed in 6.17.0-6.19.0; bump past all of them * Revert "chore(deps): bump pypdf to 6.19.0 to clear osv-scan" This reverts commit 0e2528123db7ab23304ae20f6a4c0961d0e2d073. * fix(proxy): keep LIT002 budget flat in the model search where-builder Extract the dynamic Prisma where-condition fragments into module-level helpers so every dict starts on a short line, split the litellm_params fallback lookup into a Final local, and annotate suppressions on the construct start lines the checker attributes * chore(tests): repoint unbounded-IN baseline at the extracted where-builder helper The bounded-router-id filter moved from _fetch_db_models_for_search into _not_in_router_ids, so the old baseline key is stale and the new helper site is the finding to record * fix(proxy): avoid searching encrypted deployment model names * refactor(ui): keep model routing status as a filter only * docs(api): regenerate model info sorting description * fix(api): remove remaining paused status sort references --------- Co-authored-by: thementalcoding <312162298+thementalcoding@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 23:19:59 -07:00
yucheng-berri	1f4ef4b7d1	feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs (#45804) ... * feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs When the setting is on, spend log rows store requester_ip_address as null in the column, the metadata and the stored request (which also drops client-IP forwarding headers), and IP addresses in stored error messages are masked, so the Logs page shows no caller IP. The setting is read the way the settings API validates it. Capture is unchanged, so guardrails, logging callbacks and metrics still receive the IP. Toggleable from the Admin UI General Settings. Resolves LIT-7442 * fix(proxy): mask IPv6 addresses that end or start with :: across eight groups The masking pattern allowed seven colons, so valid forms like 1:2:3:4:5:6:7:: and ::1:2:3:4:5:6:7 stayed in stored error text with disable_requester_ip_address_logging on. Also lists the request metadata walker in the recursion check, since it only walks the tree the spend log sanitizer already returned. * fix(proxy): mask IPs in every stored header value and after labels or underscores With disable_requester_ip_address_logging on, client IPs carried in headers outside the forwarding list (do-connecting-ip, incap-client-ip, custom names) and addresses written as host:2001:db8::2, peer_2001:db8::2, socket_192.0.2.1 or an unbracketed IPv6 with a port were still stored in spend logs. * fix(proxy): mask request header IPs before long values are truncated A header value longer than the spend log string limit kept the part of an IP that fell before the truncation cut. Request headers in metadata and requester_metadata are now masked before the body is truncated, like error text already was. IPv4 followed by a dotted port (192.0.2.1.443) is masked again. * fix(proxy): mask the IP piece a header truncation cut leaves instead of pre-masking headers Masking header values before truncation scanned the whole client value, including text the cut drops, and walked hook-made header cycles. Mask after truncation instead, then mask the hex and digit run touching each side of the truncation marker when it is short enough to be part of an IP address. Every nested headers mapping gets the same treatment. * docs(proxy): say the requester IP setting masks IPs in other stored header values * fix(proxy): mask stored error text after truncation and harden the cut-piece pass Error messages and tracebacks were regex-scanned in full before the DB cap, which costs seconds on multi-megabyte provider errors. Mask them after the cap like header values, with the same pass over the run beside each truncation marker. That pass now handles every marker, skips long hex tokens so an address beside one is still masked, keeps at most one address length, and also masks colon-only IPv6 groups. * fix(proxy): drop the requester IP from the originating request an auto router classifier row stores * fix(proxy): drop the requester IP from the classifier input an auto router classifier row stores * refactor(proxy): mask stored IPs with the content filter's ipv4 and ipv6 patterns The requester IP setting now reuses the content filter guardrail's prebuilt ipv4 and ipv6 patterns instead of its own address regexes, so spend logs mask what that guardrail masks. Addresses glued to a label or underscore stay as the content filter leaves them. * fix(proxy): mask stored IPs with the spend-log matcher again The content filter's ipv4 and ipv6 patterns left labelled IPv6 (host:2001:db8::2), unbracketed IPv6 with a port and IPv4 after an underscore in stored error text and header values, and masked parts of text like std::io::Error. Restore the address-validating matcher and its regression cases. This reverts commit 79537a8aa5. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-14 18:16:40 -07:00
Kerry Lu	4f34856a65	refactor(router): name the pre-content stream retry helpers for both stream paths and let the retry loop take its attempt function (#46474) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 18:25:15 +00:00
Yassin Kortam	ae9c208b8d	fix(proxy-extras): bound migration lock waits with SET LOCAL lock_timeout in the migration SQL (#46329) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-13 20:51:03 -07:00
devin-ai-integration[bot]	f3a560b73c	feat(telemetry): add litellm.telemetry records, semantic group consent and sink protocol (#45484) ... * feat(telemetry): add litellm.telemetry records, consent groups and sink protocol * docs(telemetry): reword TelemetryGroup docstring and drop module docstring * fix(telemetry): add timeout and connection_error status classes, check relative imports in the package boundary test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: nate <nate@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-13 18:19:22 -07:00
devin-ai-integration[bot]	a4382657fd	chore: drop Greptile from the PR template, AGENTS.md and CI (#46013) ... * chore: drop Greptile from the PR template, AGENTS.md and CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: require bot reviews to finish before ticking the review checkbox Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-12 18:32:39 +00:00
devin-ai-integration[bot]	4b8a846788	test: remove pass-through and user_api_key_auth coverage scripts that no CI job runs (#45969) ... Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-11 14:44:35 -07:00
devin-ai-integration[bot]	a7a3a6b375	test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) ... * test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e * test: fix disk cache collection, ungate saturation check, drop RSS memory tests * test: assert one delivered log per completion in Datadog and LangSmith checks * test: drop SDK-direct e2e copies and align cost-map path filter cases --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 06:24:57 -07:00
devin-ai-integration[bot]	bc2cc42340	docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI (#45838) ... * docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): cap _inlined schema walk depth for the recursion check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 21:04:59 +00:00
Soham Chatterjee	38bdda9ca7	feat(scaledown): add native text provider support (#44168) ... * Add ScaleDown model pricing Registers the five ScaleDown models in the pricing map. ScaleDown bills on input tokens only, so output_cost_per_token is zero for every entry. The classify and decisions models bill at $0.04 per million input tokens, and extract, summarize, and compress bill at $0.05 per million. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Price all ScaleDown models at $0.04 per million and drop unverified context limits Every ScaleDown model now bills $0.04 per million input tokens. The 32000-token max_input_tokens/max_output_tokens values were not confirmed against the service, so they are removed rather than registered. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Price all ScaleDown models at $0.05 per million input tokens This is the standard rate in ScaleDown's billing and matches the usage dashboard. The Decisions usage.cost field is not used for pricing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add regression tests for the ScaleDown pricing entries Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Cite the source of the ScaleDown rate in the pricing tests and type them Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add ScaleDown as a chat provider ScaleDown serves five models over two wire protocols. extract, summarize, and compress speak OpenAI chat on /v1/chat/completions, while classify and decisions implement the Jev Decisions API on /v1/scaledown, taking a state object plus a questions map and returning one typed answer per question. The provider authenticates with x-api-key rather than a bearer token, and the decisions models need a non-chat request body, so it routes through the HTTP handler instead of the OpenAI passthrough. Malformed decisions requests are rejected locally with a message naming the problem, so callers do not have to decode an opaque 422 from upstream. Decisions bills one model call per question, so the cost upstream reports is passed through rather than re-estimated from token counts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Route ScaleDown domain models to the native API and stop trusting usage.cost The /v1/chat/completions and /v1/models routes return 403 on api.scaledown.xyz, so extract, summarize and compress now call the native /extract, /summarization/abstractive and /compress/raw/ endpoints. The default api_base is the bare host; a trailing /v1 is stripped and re-added only for decisions. The Decisions usage.cost field was about 600x the dashboard amount, so it is no longer passed through; cost is computed from input tokens at the registered rate. Native responses are returned unchanged as JSON, including the _value/_span_anchor wrappers on nested extraction. They carry no output token count, so completion_tokens is reported as 0 and documented as unmeasured. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Move ScaleDown translation into llms/ and address review findings - Dispatch moves to llms/scaledown/chat/handler.py; main.py keeps a thin call and get_llm_provider_logic.py no longer has a ScaleDown branch, since the config resolves key and base URL itself. - questions passed via extra_body are merged and validated after the shared merge (sign_request), so the documented extra_body path works. - extra_body can no longer change the upstream model, and decisions state may carry a document but not text, so guardrails always see the text. - A custom api_base requires its own api_key; the environment key is only sent to the default host or SCALEDOWN_API_BASE. - Image content parts become the decisions document; max_completion_tokens maps to max_tokens; stream is served as one chunk for every model. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Refuse prompt-bearing extra_body keys and empty-key fallback to the env credential extra_body is merged after guardrails inspect the messages, so each model now accepts only option keys there (extract threshold/top_n, compress compression_rate, decisions questions/state); text, instructions, prompt, context and model are rejected. An empty api_key no longer lets the SCALEDOWN_API_KEY from the environment go to a custom api_base. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Read compress token usage from the real response shape /compress/raw/ nests the per-prompt counts under results and reports the total as a top-level input_tokens; the provider read a top-level original_prompt_tokens that is not there, so compress reported zero tokens and zero cost. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Accept the router's max_retries and word the extra_body error clearly The proxy adds max_retries to every call, which map_openai_params rejected, so every proxied ScaleDown request failed with a 400. It is now accepted and not forwarded. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Return extracted fields from extract, follow schema refs, and meet the type-discipline gate - extract content is now the extracted fields (span anchors dropped, wrappers unwrapped), so it validates against the caller's response_format; the raw payload stays on _hidden_params['scaledown_response']. - Local $ref/$defs in the response_format schema are resolved. - The adapter is rewritten with Final locals, read-only annotations, one raising helper and a typed handler; check_type_discipline reports no violations. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Fix stream_options, schema-guided extract cleaning, document input and multi-image handling - stream_options is accepted and not sent upstream, so include_usage streams work. - Extract output is cleaned against the requested entities, so a field named _value or *_span_anchor is no longer dropped. - Extract and summarize send an image or PDF as the native document fields. - More than one image is rejected instead of silently using the first. - The cost test derives its rate from the cost map. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Cut recursive schema refs at the cycle and cap the expanded entity count A small response_format schema with a self-referencing $def expanded to 4^16 leaves before any request was sent. Refs that point back into themselves are now cut off at that point, and the total expanded entities are capped at 1000 so repeated acyclic refs are rejected with a clear error. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Bound ref chains and schema nesting so long schemas fail cleanly Resolving a chain of thousands of distinct $refs recursed until RecursionError. Ref resolution is now an iterative walk capped at 32 hops, and nested properties are capped at 32 levels; both raise a clear 400 instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(scaledown): support native classify and preserve extraction results * test(scaledown): include provider regressions in CI shard * fix(scaledown): pass registry typing and bounded recursion checks * docs(scaledown): capture provider setup before and after * fix(scaledown): preserve nullable schemas and reject malformed requests * fix(scaledown): isolate provider dispatch from existing providers * fix(scaledown): preserve the shared dispatch return contract * fix(scaledown): preserve failure logging and reject lossy schemas * fix(scaledown): keep timeout typing annotation on its expression * fix(scaledown): resolve escaped extraction schema references * fix(scaledown): annotate the bounded reference walk * chore(scaledown): remove unused loop suppression --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: moe-berri <moe@berri.ai>	2026-10-10 14:01:17 -07:00
Mateo Wang	3c56cd31f5	chore(greptile): load nested AGENTS.md files as review context scoped to their directories (#45636) ... * chore(greptile): load nested AGENTS.md files as review context scoped to their directories * refactor(greptile): use a stdlib dataclass so the files.json check runs without uv * test(greptile): write fixture AGENTS.md files through a helper instead of rebinding a loop variable	2026-10-09 13:22:29 -07:00
devin-ai-integration[bot]	4b975e6f49	ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) ... * ci: restructure lint, unit and smoke workflows * ci: preserve source formatting in tier workflows * ci: retire per-shard coverage flags and tighten the tier workflow guards * test(ci): freeze the workflow startup safety models * ci: keep the current required check names running until the ruleset moves to the tier collectors --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-09 04:28:41 +00:00
devin-ai-integration[bot]	8a176c0f0a	fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt (#45393) ... * fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt An order-based or configured fallback hop replayed the failed provider's encrypted reasoning items to the next deployment, which answered 400 (Bedrock Mantle: invalid encrypted reasoning; OpenAI: invalid_encrypted_content), so every multi-turn Responses fallback for Codex-style clients failed. The hop now drops the encrypted reasoning its target cannot decrypt and keeps each item's readable summary. With encrypted_content_affinity on, the pin narrows to the hop's target order instead of emptying it, so the hop reaches the next order instead of failing with no deployments available. * fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt Unmarked encrypted reasoning on a hop is attributed to the deployment that just failed, read from the retry breadcrumb, so a hop to a deployment on the same api_base and api_key keeps it and a cross-provider hop still drops it. The hop tests script the upstream at the httpx boundary instead of doubling the handler, and the router coverage script lists the two hop helpers with their tests * fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot * test(router): use a real Router without the origin deployment in the hop strip test * test(router): cover the hop strip when no failed deployment is known * fix(proxy): drop the router's fallback hop state keys from the client body * fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys * test(integration): audit cells for the fallback hop encrypted reasoning strip Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 17:30:01 -07:00
devin-ai-integration[bot]	1a42c5abde	ci(unit): add unit passed collector job and fold proxy-db shards into test-unit.yml (#45466) ... * ci(unit): add unit passed collector job and fold proxy-db shards into test-unit.yml * test(ci): cover the unit passed gate's success, failure, cancelled and skipped results * test(ci): run the unit passed gate test from the code quality workflow --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 17:24:24 -07:00
devin-ai-integration[bot]	b7ecb05340	fix(proxy): mark background responses stale_expired when provider returns 404 (#41724) ... * fix(proxy): mark background responses stale_expired when provider returns 404 Responses deleted upstream (store=false / ZDR rows dropped after provider retention) now move to stale_expired on the poll instead of being retried every cycle until the 7 day stale cleanup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop inline comment in check_responses_cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive _get_response seam in 404 stale_expired tests Assigning AsyncMock on the instance keeps the new tests inside the TQ002/TQ008 test-quality ceilings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only expire 404 responses polled through a resolved deployment A NotFoundError from the bare SDK fallback can mean a missing or misconfigured deployment, which a config fix inside the staleness window can still recover. Rows whose poll went through their router deployment are the ones the provider actually dropped, so only those move to stale_expired. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve the deployment before fetching so via_router binds once Drops the reinitialised loop flag flagged by review and modernizes the moved annotation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive the real response fetch in the poll 404 regression tests * fix(enterprise): expire background response rows on any provider 404 status The GET path maps the provider's 404 body to litellm.BadRequestError that still carries status_code 404, so an except on NotFoundError never fired and the poll job kept retrying the row every cycle. Key the terminal decision on the status code instead of the class * fix(enterprise): expire a background response only on a 404 naming it and skip a bad deployment entry per job * chore(enterprise): record the poll-cycle bound on the managed-object IN lists * test(integration): cover background response poll retirement on the real proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 10:35:52 -07:00
Mateo Wang	58258409c9	refactor(proxy): expose public names for private proxy helpers (#45170) ... * refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 18:59:27 -07:00
RachelHuangZW	bf9f59d76b	fix(scheduler): remove a request's queue entry once it stops waiting (#43061) ... * fix(scheduler): remove a request's queue entry once it stops waiting Requests admitted while a healthy deployment existed never left the priority queue, and neither did requests cancelled while waiting. The stale entries blocked later requests during cooldown and, with Redis, made add_request raise a TypeError on queues read back as JSON lists Both scheduling paths now share one polling helper that removes the entry in a finally block, whether the request was admitted, timed out or cancelled Related to #43059 * test(router): allowlist _wait_for_scheduler_turn in the router coverage check The coverage script only counts direct calls in test files. The helper is exercised through prioritized acompletion and atext_completion in test_router.py, like the other allowlisted entries * fix(scheduler): admit healthy requests before reading the queue, and clean up cancelled enqueues poll() raised on an empty queue before checking for healthy deployments. With the cleanup now rewriting the queue after every admission, a concurrent write from another replica can erase a waiting request's entry, and that request then failed while a deployment was healthy. poll() now admits as soon as a deployment is healthy and only reads the queue during cooldown add_request also moved inside the try block, so a request cancelled while its queue write is in flight still has its entry removed * refactor(scheduler): move the wait loop into Scheduler.wait_for_turn The router passes a healthy-deployments callable into the scheduler, so tests inject a Scheduler directly instead of replacing the router's scheduler attribute. The cancelled-mid-enqueue test moves to test_scheduler.py with the other scheduler tests, and poll() takes the deployments as a Sequence since it only checks whether any are healthy * test(scheduler): move scheduler tests into tests/unit * fix(scheduler): finish the queue removal when cancellation is delivered again during cleanup A second cancel, or an anyio cancel scope that re-cancels on every await, interrupted remove_request mid-write and left the entry in Redis. * fix(scheduler): read queue entries back from redis as tuples * fix(scheduler): admit a request whose queue entry vanished during cooldown * fix(scheduler): re-enqueue a request whose queue entry vanished during cooldown * test(integration): cover priority scheduler queue cleanup across instances Adds tests/integration/routing/test_priority_scheduler_queue_cleanup.py: 27 cells against the real proxy (two workers, Postgres, Redis, a scripted upstream) for every prioritized surface (/v1/chat/completions, /v1/completions, /queue/chat/completions, streaming and not, OpenAI SDK sync and async, raw httpx), the non-integer priority pass-through, the in-memory queue on one proxy, two proxies sharing a Redis queue (served requests leave no entry, a dead replica's entry is skipped or expires, a waiter that times out or disconnects removes only itself), a Redis outage mid burst and a SIGKILLed worker. Each cell asserts the caller's response, the upstream's requests by marker and the Redis queue contents. On the merge base the cross-instance cells fail with the list-of-lists TypeError and the in-memory cell with 408s behind the leaked entry; on this branch every cell passes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 17:34:38 -07:00
moe-berri	e9cfba2c17	feat(lens): isolate ingestion and investigations in a Rust service (#45148) ... * feat(lens): isolate trace storage and investigation in a Rust service * fix(lens): include Rust sources in the image build context * feat(lens): wire service setup, scoped delivery receipts and lease attempts * fix(lens): complete service routing and reject stale investigation results * fix(lens): retry key propagation and validate isolated Compose setup * fix(lens): seed through isolated ingestion and preserve upstream queue fixes * chore: sync schema.prisma copies from root * fix(lens): bind nullable due timestamps as text for Prisma * chore(ui): remove stale lint suppressions * fix(lens): address CI failures and review findings * refactor(lens): remove retired Python worker and run evaluations in Rust * fix(lens): reuse control connections and satisfy review checks * test(lens): install and upgrade both Helm charts on Kubernetes * test(lens): run connection reuse coverage as an integration test * fix(ui): upgrade Next.js to 16.3.8 security release * fix(lens): fence stale attempts and preserve reviewed evidence * Revert "fix(ui): upgrade Next.js to 16.3.8 security release" This reverts commit 2f79a51b25. * fix(lens): stop failed investigations and stream history excerpts * test(lens): cover model tool and result contracts * test(lens): fix retired routes and reuse installation build artifacts * test(lens): use portable grep in Helm installation smoke * test(lens): wait for migrations before forwarding Helm services * fix(lens): keep failed evidence reads retryable * fix(lens): preserve sandbox output during process exit --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-10-07 17:18:58 -07:00
ryan-crabbe-berri	679f7e636e	test(e2e): record steps for raw transport calls and poll helpers (#45150) ... Add @step labels to the HttpTransport methods, the poll and wait helpers and the boot helpers that did real IO without recording a step, so a test that reaches the proxy through them no longer reports an empty or gappy step timeline in the JUnit report.	2026-10-07 15:21:26 -07:00
ryan-crabbe-berri	048a1500df	test(e2e): move the harness self-tests out of tests/e2e (#45172) ... * test(e2e): move the harness self-tests out of tests/e2e The nightly Buildkite run copies tests/e2e into the runner image and runs bare pytest, so the 672 tests of the harness itself (fixture parsing, JUnit properties, the stack lock, the load aggregators, the Claude Code driver) counted as e2e tests on the status page even though none of them reaches a proxy. They now live in tests/e2e_harness, mirroring the tests/e2e layout, and run in the GitHub Actions lint job and the CircleCI provider_replay_harness job instead * fix(ci): point the providers replay controls at tests/e2e_harness The providers integration job still selected the four replay-control tests under tests/e2e/test_provider_edge.py, so pytest exited before they ran. The raw-HTTP check's file walk also drops to one loop per comprehension * style(tests): mark the raw-HTTP check's bindings Final	2026-10-07 22:20:02 +00:00
tin-berri	0338498067	fix(router): preserve native baseline identity and accounting (#44960) ... * fix(router): preserve native baseline identity and accounting * fix(router): preserve injected system caches in native baselines * fix(router): prepare native baselines through shared request owners * fix(router): capture native baseline fields from the provider schema * refactor(router): reuse native provider parameter discovery * fix(router): keep long native baselines and abstain after compaction Message history no longer counts against the settings snapshot budget, so long and non-ASCII native sessions keep modeled baselines. Selected-tier compaction now abstains because the baseline would otherwise inherit the compacted history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): judge implicit caching against the selected request The implicit-cache guard compared the selected response's cache usage with the projected baseline's breakpoints, so selected-tier cache markers made a usable unmarked baseline plan look like unexplained caching. The guard now checks the selected wire request. Also removes a stamp-reuse branch that could never run because routing clears the stamp first; every pass already captures caller settings from fresh kwargs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-07 11:00:31 -07:00
devin-ai-integration[bot]	e2971e0af4	refactor(llms): expose public names for private provider helpers (#45037) ... * refactor(litellm): migrate private usage in llms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve Bedrock batch signature marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): migrate llms private usage symbols Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve UUID Watsonx project IDs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): retarget llms mocks to public names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): limit llms changes to renames and forwarders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): keep GCS mock client patching private Vertex auth methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): assert forwarder arguments and type forwarder signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 08:49:25 -07:00
devin-ai-integration[bot]	dfe4df8c8a	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) ... * test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-07 05:02:28 +00:00
devin-ai-integration[bot]	c3a23fe499	refactor: expose core private helpers under public names (#44871) ... * refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 00:05:27 +00:00
ryan-crabbe-berri	0b0fdedd1e	test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) ... * test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata * docs(e2e): name every markerless harness test file that carries no Subject * test(e2e): keep the step discovery comprehensions to one for clause	2026-10-06 17:03:35 -07:00
devin-ai-integration[bot]	3fec4705d7	fix(projects): show Projects to team and org admins and scope it correctly (#41325) ... * fix(ui): show Projects nav to team and org admins Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): satisfy frontend lint budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format leftnav regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(projects): let org admins see projects of every team in their orgs /project/list and /project/info only knew proxy admins and team members, so an org admin saw projects only for teams they had personally joined. Both now use the same team access check as /team/info. * fix(ui): one shared Projects access rule for nav, data and actions The Projects query skipped global org_admin users, the page-visibility picker could never offer Projects, and New/Edit showed to users the backend would reject. Nav and queries now share one rule, Projects is selectable in the allowlist, New/Edit follow team_admin_editable_team_fields, and the project modal only lists teams the user administers. * fix(projects): record IN-list bounds for project visibility filters Both filters are bounded by one caller's team memberships and admin orgs. Also cover a project whose team was deleted. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-10-06 14:33:31 -07:00
devin-ai-integration[bot]	5cdebded90	fix(ci): skip the generated dashboard bundle in the master key guard (#44890) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 18:15:09 +00:00
devin-ai-integration[bot]	837c6a7481	fix(security): remove the publicly known master key from the repo (#44718) ... * fix(security): hash the publicly known master key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: replace weak master key examples and regenerate artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: replace weak key fixtures with generated test keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: generate master keys for proxy startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve lens dev key entropy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore proxy key compatibility in scrub examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: scrub merged SSO fixture and refresh dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): stabilize test keys and metadata collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 10:55:24 -07:00
devin-ai-integration[bot]	c13746c951	fix(router): price a model group from the deployments that serve it (#44732) ... * fix(router): price a model group from the deployments that serve it A model group's info read composed the group's own model_group_alias entry into its deployments, a hop the router never takes: an alias is resolved exactly once at request time, so a group reached as an alias target is served by its own deployments. For the chain X -> T -> U the price read for X included U's deployments too, and the free-model budget waiver refused a free request to X on an over-budget key, while GET /model_group/info reported U's providers and price for X. The group info read now prices a group from the deployments routing serves it with: the ones named after it, the routing group of that name, or the wildcard route matching it when neither exists. The budget waiver, GET /model_group/info, the rate limiters, and the response headers all read the same set as routing. get_model_list keeps its behavior for every other caller. * test(router): give the paid fixtures explicit per-token prices * test(router): call the routed-group read by name so the router coverage gate sees it * test(integration): audit the alias chain budget waiver on every route, shape, and outage Thirty-four cells under the management group prove an over-budget key is served through an alias chain entry at the price of the deployment that serves it, on chat, responses, and messages, sync and streamed, through the OpenAI and Anthropic SDKs and raw httpx on both replicas, with the chain middle, the reverse chain, a cost-map priced middle, a ghost middle, wildcard and routing-group targets, malformed and hostile inputs, a cached reply, a repointed alias, a provider failure, and two chaos bursts (a killed worker, a scripted outage) --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-06 05:31:09 +00:00
devin-ai-integration[bot]	d7f69d6ba1	refactor(logging): load enterprise alerting loggers lazily so import litellm skips proxy types (#44762) ... * refactor(logging): load enterprise alerting loggers lazily so import litellm skips proxy types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): cover lazy alerting dispatch and docs lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve litellm.proxy._types lazily on attribute access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(imports): assert proxy types guard imports the checkout under test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: nate <nate@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-05 22:06:53 -07:00