* feat(sso): create the team_id_jwt_field team on UI SSO login when team_id_upsert is set
* fix(sso): keep the upserted team_id_jwt_field team out of result.team_ids so restricted_sso_group and the access-token group fallback are unchanged
* fix(sso): only upsert the team_id_jwt_field team on UI SSO login when enable_jwt_auth is on
The API JWT path only reaches team_id_upsert inside the enable_jwt_auth branch, so a deployment
with a leftover litellm_jwtauth block and JWT auth off never created teams. Gate the UI SSO upsert
the same way
* fix(sso): read enable_jwt_auth through the proxy_server module so the basedpyright gate stays at its ceiling
* fix(proxy): evict a deleted key from every worker's auth cache
/key/delete dropped the key object only from the in-memory cache of the worker
that served the call. Auth serves key objects cache-first, so every other
worker, and the Redis copy when enable_redis_auth_cache is on, kept
authenticating the deleted key until user_api_key_cache_ttl expired. /key/update
already evicts and broadcasts through evict_and_broadcast, delete now does the
same for the tokens it deletes
* fix(proxy): evict only the deleted rows and broadcast them in one batch
Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker
* refactor(proxy): drop the comment above the delete eviction call
* fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait
* test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests
* test(proxy): wait for the peer to apply the broadcast instead of polling its cache
* test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer
* fix(key_management): evict every requested key on delete, including keys whose row is already gone
A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone
* test(key_management): split the gone-row eviction assertion into one comprehension per worker
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): name shared-handler errors by the status they answer
handle_exception_on_proxy pinned type=internal_server_error on every
exception it wrapped, so a management route rejecting a caller's own
argument answered {"type":"internal_server_error","code":"400"} and an
SDK branching on the type read its own bad request as a gateway outage.
Derive the type from the status the response actually carries, keeping
a type the exception already names.
* test(proxy): pin a 500 for exceptions whose status is None
* fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors
Provider 5xx and timeouts keep internal_server_error so retry loops still retry them,
the shared handler redacts internal hosts from raw exception messages, and /key/list
routes its HTTPException through the shared handler instead of its own copy of the
old label.
* fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures
* fix(proxy): pass shared-handler error messages through as on main
Redacting timeout and 5xx messages changed what every shared-handler route
returns, which relabeling the error type does not need, so messages go back
to the verbatim text main sends.
* fix(proxy): redact only the status-less failure main never sent
* fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403
A failure with no status now answers "Internal server error" instead of its own text, since that text can
name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The
/user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises
* fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise
A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces
* refactor(proxy): parse a three-digit string status without recursing
* fix(proxy): apply published model rows on every worker before the debounced resync
Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path
* fix(proxy): load published model rows from the primary under the reconcile lock
Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s.
* test(proxy): cover a published model row that names no credential
* fix(proxy): keep the unknown model name read-through on the replica and outside the lock
Only the by-id load that follows a published write reads the primary inside
the reconcile lock. A request for a model name a worker does not know reads
through the replica routing before taking the lock again, as on main, so
unknown names put no load on the primary and do not wait behind a reload.
* fix(proxy): keep access group edits from dropping staying deployments and batch published config ids
* fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica
* fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing
A batch of published model ids that held one auto router skipped every row in it, so ordinary
rows written at the same moment waited for the full reload. The by-id load now leaves only the
auto router rows to that reload.
An access group update now reads every deployment from the primary, answers 404 for an unknown
group and 400 for an unknown model id or name before it writes anything, then removes the group
from the deployments that leave before it adds it to the ones that join, so a failed write leaves
the group narrower than asked, never wider.
* fix(proxy): write an access group update in one transaction after a fresh read
The update now checks model names first, reads every deployment from the
primary, and commits every join and leave in one batch right after that
read, so a failed write changes nothing and a field another admin set
during validation is kept. Other workers hear about the rows only after
the commit.
* fix(proxy): lock the model rows while an access group update reads and rewrites them
* test(proxy): run the model resync tests on the real installer and a real router
The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen.
* fix(proxy): lock only the rows an access group update touches and answer 404 before 400
An access group update locked every model row, so a lock on any unrelated
row stalled it for 30 s and then failed it with a 500, and unrelated model
edits on other proxies waited behind it. It now takes a per-group advisory
lock, locks only the rows that carry the group or that it selects, and
reads them again once locked. The group existence check runs before the
model name check again, so a missing group answers 404 as on main.
* fix(proxy): never wait on a locked row in an access group update
The update now takes its row locks with NOWAIT. When another edit holds one
of the group's rows, it rolls back, holding nothing, and tries again for about
10 s before answering 409. Holding rows while waiting deadlocked against team
deletes and let a concurrent model edit undo the update's change. The
per-group lock also moves to the two-key advisory space, apart from team ids.
* fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select
* fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
* fix(fallback): read stored router_settings from the database on fallback writes
POST /fallback and DELETE /fallback/{model} read the current rules through the
config cache. When Redis refuses DEL, the eviction before that read fails and
Redis keeps serving the old copy, so each new rule dropped the earlier ones
The write path now reads the router_settings row straight from the writer
database and stores only that row plus the changed key, so config-file router
settings no longer get copied into the database row
* fix(fallback): edit the config file's fallback list when it owns the key
A delete of a rule only the config file holds answers the config-owned 400 again instead of 404, and restating a rule the file already has answers 200 without copying the file's list into the database row. Adds a writer-read regression test and an integration test against a Redis whose ACL refuses DEL
* fix(fallback): re-read the config file's fallbacks ownership on every fallback write
* test(fallback): run the real config reload in the fallback write tests
* test(fallback): key the refused-DEL Redis double by cache key
* fix(fallback): keep the config-owned 400 for an unset env reference and replace a non-object stored row
* fix(fallback): replace a stored router_settings row that holds a non-JSON string
* fix(fallback): compare a restated config-file rule against its resolved env value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: stabilize cleanup, lockout, and post-kill timing tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bound the wedged Redis burst p95 by the socket timeout the test sets
The burst p95 under a fixed 0.75s cap tracked how fast the box ran the 30
concurrent writes, so a loaded runner failed it at unchanged code. A handler
that waits on the wedged publish takes at least the coordination Redis socket
timeout, so the test now sets that timeout and asserts the p95 stays under it.
* test: pin the lockout clock and bound the wedged Redis burst by the proxy's 2s wait cap
The lockout tests now set the login throttle's clock instead of reading the real one, so Retry-After is pinned at 77 and a round-down fails on every run
The wedged Redis burst p95 is now bounded by the 2s cap the proxy puts on its own Redis waits, so a burst wait capped under the 5s socket timeout still fails the test
The fresh post-kill clients keep the harness's 2s keep-alive expiry, here and in the two tests #47080 changed
* test: tell ceil from round in the lockout tests and bound the burst by the imported eviction cap
The refused attempt now runs with 76.4s left on the 77s block, where ceil gives 77 but round and int give 76, so
both lockout tests fail if Retry-After stops rounding up.
The burst bound imports _BULK_EVICTION_WAIT_SECONDS instead of copying its 2s value, so lowering the cap moves
the bound with it.
* test: bound the wedged Redis burst by the shorter of the eviction cap and the socket timeout
Any wait the proxy puts on the stopped Redis lasts at least the shorter of the two, so the bound now follows
whichever one a later change lowers.
* test: drive the tool index cleanup test with the manual clock and keep the post-kill block as on main
The post-kill reset is already fixed on main by making kill() wait for the
worker's last thread, so the fresh-client and keep-alive edits go back to
main's version. The tool index cleanup test now advances the injected clock
instead of timing the real one, and the lockout clock helper gets a type hint.
* test: advance the spend log cleanup tests' manual clock through its own method
* test: freeze the throttle clock in the login_utils lockout tests and drive the run budget tests with the manual clock
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover management and health routes labelling rejections as auth failures
* fix(proxy): name management and health route errors by their status instead of auth_error
* fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls
/key/list and /key/aliases now share the catch-all helper too, so they stop
adding the Authentication Error prefix and typing a 400 as internal_server_error
* fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error
A real outage surfaces from prisma's query engine as a builtin AttributeError, which the
prisma-module check skipped, so the route catch-alls still answered 500 or 400
* fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status
* fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503
* fix(proxy): answer 503 for database failovers and label provider auth errors like chat
A database that refuses connections while starting up, shutting down, in
recovery, or read-only after a failover now answers 503 no_db_connection
instead of 400. A provider's 401 or 403 keeps the type /chat/completions
gives it, while a 401 or 403 the proxy raises keeps auth_error. An
exception with no message is named by its type.
* fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause
* test(proxy): expect the OpenAI error shape from management route refusals
Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(scim): audit #47544
Add integration cells for the SCIM group delete that now runs through the team delete's bulk path: an admin group delete revokes its members' admin key, a delete sent with the IdP's SCIM token revokes member keys and audits them under that token, a delete that times out on a locked member row changes nothing so a retry works, and two racing team deletes write one tombstone and one team audit row
* test(scim): read the racing team delete's audit rows after the proxy drains them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(teams): show caller member spend and budget on Your Teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): use member-linked reset dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep type-discipline suppressions on cast lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): preserve empty page list response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep the member reset date inside the spend column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): match email-only roster members for caller membership
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show team member budget reset date in UTC
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(teams): address review on caller membership enrichment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): label the caller's team spend line My Spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): sort every teams table data column
* fix(ui): order unlimited team limits as infinity and verify org scope
* fix(ui): validate team sort filters without recursion
* feat(ui): sort every users table data column
* fix(ui): order unlimited user budgets as infinity and verify org scope
* fix(ui): cover user sort queries and document pagination bound
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): delete a group through the team delete's bulk path instead of one member at a time
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(scim): demote a deleted admin group's members before the team delete so a failed delete stays retryable
* fix(scim): recompute member roles again after the team delete so a concurrent group write cannot regrant admin
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(scim): write recomputed member roles only where the membership still backs them
* fix(team): report the users a team delete detaches so SCIM demotes late-added members
* test(scim): audit cells for the bulk group delete
Move the SCIM group helpers into tests/integration/_support/scim.py and add the delete cells: a member's other
group survives, the admin group's members are demoted and the audit row names the SCIM caller, a member a
concurrent PUT adds while the delete waits on the team lock is demoted too, unknown, repeated, non-admin and
hostile ids, an empty group, the peer proxy, a 20-delete burst across both proxies, and a delete after a worker
was SIGKILLed
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(team): sweep deleted team references only under the lock so a late-added member is demoted
The unlocked sweep that ran before the team's advisory lock stripped a member a
concurrent group write added after the roster was read, so neither the roster
snapshot nor the locked sweep's RETURNING named them and the SCIM recompute
left them proxy_admin after the admin group was gone. The locked sweep does the
same strip and reports whom it stripped, so it is the only sweep now.
The integration cell holds the delete on a member's user row so a PATCH lands
between the roster read and the lock, then checks the late member is demoted.
Both waiter queries in the SCIM support module return their count as text so
the first poll parses.
* chore(team): carry the unbounded-IN baseline entries to delete_validated_teams
The delete pass moved from delete_team and the unlocked sweep into delete_validated_teams, so the checker's baseline names the function the two team_id IN lists now live in
* test(scim): expect the master key alias in the audit row and send post-kill requests on a fresh connection
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* test(scim): cover deleting a SCIM group through the team delete's bulk path
* fix(scim): evict a deleted group's member caches in bulk and drop cached roles after a recompute
* fix(scim): answer 404 to a group delete that lost a race to another delete
* fix(scim): chunk the locked team lookup and delete in the team delete transaction
* fix(scim): keep service-account keys and team models on a group delete and write its records once under the lock
* fix(scim): demote a deleted group's members inside the delete transaction
The admin-group role write now runs on the team delete's transaction, over the
roster read under the team lock, so a failed delete leaves roles untouched and
a member added while the delete waited is demoted with the rest. The delete
transaction also waits for a free connection as long as it may run, so a burst
of deletes queues instead of failing after Prisma's two-second default. Every
member in the 5-vs-500 statement test now holds a key.
* fix(audit): write deleted keys' audit rows in one insert
A delete that removes many keys wrote one audit insert task per key, and those tasks held pool connections the delete's follow-up writes waited on. The race test now waits on any lock wait behind the delete.
* feat(model hub): show admin-defined custom tags and metadata from model_info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): move model hub custom field merging into router_utils
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): share the custom tag badge list between the column and details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(model hub): cover public hub, patch, null, hostile, forwarding, and killed-worker paths for custom fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): drop custom_metadata entries with non-string keys or non-finite numbers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): use isinstance narrowing instead of TypeGuard for custom_metadata values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): build custom tags and metadata in the proxy instead of the router
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): serve custom tags and metadata on model_group_alias rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remove docstring from model_group_custom_fields.py
* style(model hub): fix lint and format in custom metadata tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(model hub): drop leading blank line in custom fields helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stabilize auth and team delete checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make loop-blocking and tampered-signature checks deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): isolate model-cost state and synchronize loop progress
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): fail loop-progress handshakes when the blocking call never starts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(key_management): apply default_key_generate_params.allowed_routes on key generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(key_management): avoid mutating default routes payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): apply default allowed_routes to key_type presets it covers
* fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy
* fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes
* fix(proxy): keep owners from changing routes a key got from default_key_generate_params
* docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off
* test(proxy): type the default_key_generate_params allowed_routes tests
* fix(proxy): let key owners send unchanged allowed_routes back on /key/update
* fix(proxy): ignore a route default with a non-string entry and match resent routes exactly
A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths.
* test(proxy): cover a non-string allowed_routes default in the key integration tests
* test(proxy): read key rows from the owned proxy's database in the default routes integration tests
* fix(proxy): leave allowed_routes out of a key update that resends them unchanged
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): refresh cached team on /team/block and /team/unblock
Both endpoints wrote `blocked` to Postgres and returned without touching the
management-object cache. Auth resolves teams through `team_id:{team_id}`
before reading the DB and `common_checks` enforces `blocked` off that cached
object, so a team whose traffic had already warmed the cache kept serving
requests after being blocked, and kept getting rejected after being
unblocked, until the entry expired.
`_refresh_cached_team` already exists for exactly this and its docstring says
every endpoint mutating `litellm_teamtable` must call it; these two were the
only mutating endpoints that didn't. The update now also asks for
`object_permission` so the refreshed cache entry doesn't drop the relation,
matching `team_model_add` and `team_model_delete`.
* fix(proxy): keep a team block from being undone by in-flight spend writes
update_cache read the cached team when a request finished and wrote that
whole copy back in a background task, so a team blocked or invalidated
while requests were in flight came back unblocked on every busy worker
until its 60 s cache entry expired. The team spend now goes onto the team
held in this worker's memory at write time, and a missing entry is left
missing for the next auth read to load.
Block and unblock responses keep litellm_model_table and object_permission
null, as before this PR, while the cache still gets both relations.
* test(proxy): type the team spend cache test parameters
* test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock
---------
Co-authored-by: Hamza Shah <hamzasepal@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): stop GET /__observations from draining the upstream log
GET /__observations used to empty the queue, so a second read in a test
saw nothing and a "never reached upstream" check could pass whether or not
the request was forwarded. GET now returns every request since the last
reset and DELETE /__observations is the reset. An autouse fixture resets
the worker upstream before each test, which also covers tests that read
without the gateway fixture.
Discarded reads used as resets become DELETEs, helpers that accumulated
drained reads now read one snapshot, and tests with several phases reset
between them. The three vacuous negative checks (rag allowlist, guardrail
opt-out chaos, xAI batch output) now assert the absence on the same read
that shows a request which did reach the upstream.
* style(integration): space the observations unit test like its neighbours
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
Both interactive transactions in /team/delete used Prisma's 5s default.
A lock wait or a short database stall past 5s expired the transaction
after the keys were already deleted, so the call returned 503 and left
the team row, memberships and team models behind
* fix(openai): route gpt-transcribe to GPT audio transcription config
* test(openai): move gpt-transcribe tests under tests/unit
* fix(openai): bill gpt-transcribe from provider usage seconds when duration is unknown
Fall back to the provider-reported usage.seconds when neither a measured
file duration nor a response duration is available, so json responses
for undecodable uploads are no longer billed at zero. Move the routing
test next to the other provider config tests and replace the structure
test with a behavior test.
* fix(openai): bill provider usage seconds only for gpt transcription models
* fix(cost): decide provider-billed transcription seconds from the deployment model, not its pricing id
* test(integration): cover gpt-transcribe default request, provider-billed spend, and health probe
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
6fb24d1d read ptu_shares through the pricing terms so a termless map reserved
nothing, but with the flag off the router skips ptu_config_error for config.yaml
entries, so such an entry loaded and served every team, against the rule that a
declared split restricts access whether or not cost attribution is on.
_deployment_shares reads the declared map again, and the router now raises
ptu_config_error for a config-sourced entry declaring ptu_shares regardless of
the flag, so a malformed split is refused at boot instead of being honoured or
ignored silently.
* fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling
A name served by a wildcard route now resolves to the shared deployment behind it, so the
limiter's ceiling and the team daily activity sizing both find it. Pass-through responses
settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a
reservation taken at admission is settled even when the success metadata lost the team.
* fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard
A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids.
* test(router): pin the wildcard route precedence that PTU sizing and the router share
* fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does
A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was
refused with a 422 and dropped at load. Fractional, infinite, boolean and string
shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix,
zero-token Responses usage, fractional-count and sizing-warning paths.
* fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move
* fix(proxy): write each PTU deployment day's team charges in one transaction
The backfill counts a deployment day as priced once any team's row exists for it,
so a rollup that wrote team A's share and lost team C's left team C unbilled for
good. Each deployment day's team rows now go through one Prisma batch with the
retry around the whole batch, so a day is fully written or not written at all and
the next backfill prices it for every team.
* fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes
* test(router): list the cached PTU ceiling helpers in the router coverage check
* fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force
* fix(proxy): bill a rerun PTU day to the current share holders alone
* fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list
* fix(proxy): read the PTU flag from the secret manager first, like get_secret
* fix(router): scope retry health lookups to the caller only for PTU share holders
* fix(router): move the share-holder retry check into ptu_shares
Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them
* test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* feat(proxy): report when each per-model budget window resets
model_max_budget_usage on /key/info, /user/info and /team/info now carries budget_reset_at per model, the open window's start plus its period, or null when no window is open. The window start is read in the same batched lookup as the spend counter
* fix(proxy): count a monthly budget window from its own start when reporting its reset
duration_in_seconds measures Nmo from now, so a window opened on January 15 and read on February 10 reported February 12, then null while the counter still refused requests until February 15. duration_in_seconds_from measures the month from the window start, and duration_in_seconds now shares that month arithmetic
* fix(proxy): measure months from timezone-aware datetimes and assert whole usage values
datetime.fromtimestamp without a tz tripped DTZ006. The month arithmetic now runs on an aware local datetime, which keeps the same wall-clock result. The duration tests move to parametrized functions, since subTest payloads carrying datetimes cannot cross the xdist worker boundary, and the new usage tests compare the whole entry
* fix(budgets): end a monthly budget window a calendar month after it opens
Each charge recounted the month from the charge date, so a monthly window
opened in a month of another length expired days before or after the reset
/key/info reports. The window is now measured from its start, by the same
helper the usage report uses. A period ending after year 9999 reports no
reset instead of failing the read.
* fix(budgets): count monthly budget windows in UTC for every router budget
Months were counted on the host's local calendar, so replicas with different TZ
settings disagreed on when a monthly window ends. Provider, deployment, tag and
per-model budgets now all measure a window from its start in UTC, and a test
covers the router budgets. A reset past year 9999 is documented as null.
* test(budgets): cover monthly tag budgets in the router window test
* test(budgets): add integration cells for budget_reset_at and monthly budget windows
* test(budgets): find a monthly window in another span on every calendar day
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(auth): return a generic 401 for an unknown key
The wrong-key 401 shipped the internal verification table name and the
submitted key's full SHA-256 hash to any unauthenticated caller. Both now
go to the proxy's own log and the response carries only "Invalid API key
provided."
The Admin UI detected a dead session by string-matching that message, so it
now reads the structured error type instead.
* Update ui/litellm-dashboard/src/lib/http/client.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(auth): log the missing key hash where every resolve path passes
Review catch: JWT key-mapping misses resolve through IdentityStore without
touching the main key path, so logging the hash there left those callers with
nothing but the generic 401. Log it at the store's raise site instead, which
every resolve path shares, and drop the now-duplicate log.
* test(auth): inject a key-less client instead of patching the DB fetch
The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect.
Hand IdentityStore a client whose get_data returns no row instead, so the miss is
driven through the real lookup rather than around it.
* fix(auth): log the unknown key's hash once on the auth failure line
The per-lookup warnings in get_key_object and IdentityStore logged whatever value
they were handed, which on the failure-logging path is the raw key a caller sent
(a provider key included), let a newline in ?key= forge log lines, and fired on
lookups that expect a miss. The hash now rides KeyNotFoundError onto the single
existing auth failure log line next to the requester IP, with control characters
escaped. Also drops the new comments and updates skipped tests that quoted the
old message.
* test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma
* test(integration): expect the generic wrong-key 401 in the management lifecycle tests
* fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite
* test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill
* test(auth): cover the realtime upgrade refusing an unknown key without echoing it
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): let proxy admins open team member defaults to team admins
* fix(proxy): keep a team admin's never-resets duration on a team's first member budget
* test(proxy): pin the existing-row and stored never-resets cases for member defaults
* feat(proxy): let proxy admins open soft_budget to team admins
* feat(proxy): let proxy admins open team metadata to team admins, keeping proxy settings with proxy admins
* test(ui): name expected objects in team admin edit access tests
Keeps local/no-large-inline-object-arg within its eslint budget
* fix(proxy): compare team metadata without recursion and count key presence
A proxy-controlled metadata key that appears or disappears with a null value
is now a change, and the JSON comparison no longer recurses
* fix(proxy): compare team metadata numbers exactly
Numbers are compared as decimals, so integers past float precision and close
floats are no longer treated as equal
* feat(proxy): let proxy admins open team member defaults to team admins
* fix(proxy): keep a team admin's never-resets duration on a team's first member budget
* test(proxy): pin the existing-row and stored never-resets cases for member defaults
* fix(proxy): evict the cached user on every proxy for every /user/update and /user/bulk_update
Previously the cached user row was only evicted and broadcast when the
update touched max_budget, model_max_budget, tpm_limit, rpm_limit or
metadata, so a models or user_role change kept being served by peer
proxies until the cache TTL expired
* test(proxy): cover user cache eviction past the first bulk invalidation batch
Drain the worker's fake-upstream request log when a test's gateway fixture starts, so traffic left by earlier tests on the same xdist worker no longer shows up in a test's first observation read
Generate the fine-tuning rig's provider file and job ids per fixture build, so a rebuilt module fixture in the same worker no longer collides on the managed object unique constraint
Wait for every wedged team_id and team_alias invalidation publish to fail before taking the recovery marker, now that team updates publish both keys
Pin the cooled-down priority scheduler request to the connection that observed the cooldown, since a sibling worker reads cooldowns from Redis on an interval
Simulate upstream outages and the held budget peer on ports the test keeps owning, instead of releasing a port another worker can bind
Start a peer proxy for the accounting group, which has a test that needs one
* feat(claude-code): emit installationPreference in marketplace.json entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): satisfy marketplace CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): qualify marketplace entry as read only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): suppress typed marketplace assembly warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): restrict installation_preference to the documented values
Accept only available, auto_install, or required so a typo returns 422 instead of being silently dropped, and cover the PUT full-replace clearing behavior with a test
* fix(claude-code): echo installation_preference on the public skill hub
* refactor(claude-code): read installation_preference with one membership check and cover hand-edited rows
* test(integration): audit installation_preference on the Claude Code marketplace
* test(integration): kill a serving worker in the marketplace chaos cell
* test(integration): give the marketplace worker-kill cell the two-boot timeout
* fix(proxy): keep installationPreference on the MarketplacePluginEntry model
* fix(proxy): refuse self-installing preferences off sha256 archives, accept the camelCase key, and echo the preference on POST and PUT
* fix(claude-code): serve installation_preference only on archive plugins and refuse both key spellings
Claude Code 2.1.75 and older reject a marketplace.json once any plugin carries installationPreference,
and every Claude Code release before archive support rejects a catalog with an archive plugin. Requiring
an archive source for every value, available included, keeps the field off catalogs those clients can read.
A request that sends installation_preference and installationPreference together now gets a 422.
* fix(claude-code): read the preference spelling that carries a value when the other is null
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): cover keys_auth endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover hashed, path and token spellings for keys_auth contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen JWT mapping lifecycle coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover reset_spend over plaintext and hashed path spellings with reset_to 0 and 0.02
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop Final in reset loop and use drain-then-fresh-refusal in warmed-peer delete proof
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): disable model info refresh in JWT lifecycle rig
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop unused discovery constant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover cache invalidation and key paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
* test(integration): cover teams endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): address review on teams contract tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drain shared upstream observations at fixture setup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove team block, member and bulk permission contracts at runtime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert email admin adds and whole membership rows for team members
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drive key update grant and revoke, Z-suffixed temp budget expiry, link LIT-9257
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close review gaps in teams endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): harden team member endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover guardrails_policies endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): let the admin submissions list carry other rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close guardrails policies review gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover guardrail auth and policy attachments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tidy guardrail policy test helpers
* test(integration): keep plain non-admin submission refusal live
* test: expand policy attachment integration coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): accept the intended 403 for plain submission keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin multi-tag header and spend row tags for policy tag scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the duplicate pydantic import left by the main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <noreply@devin.ai>
* test(integration): cover vector_stores endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen vector store route contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): update vector store route contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop owner-update BUG skip for admin-only vector store routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector_stores): cover the managed-file fallback, owner delete and per-request retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): probe vector store worker readiness by deployment id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin vector store retry header and create routings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(integration): ruff format vector store managed file tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
