Keeps main's Path import next to the branch's Final import in the batches endpoint tests, and takes the integration test's closed port from allocate_port so it follows main's TQ012 rule
* fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache
SAML login tickets, the single-use assertion mark and cached IdP metadata
lived in user_api_key_cache, which only gets Redis when
enable_redis_auth_cache is on. With several replicas or workers the IdP
callback often landed on a process that never saw the ticket and got a
401, and IdP-initiated sign-in was refused even with Redis configured.
SAML state now has its own DualCache that _attach_redis_usage_cache
always wires to Redis, like the CLI SSO session cache, and the SAML cache
calls are async.
* test(sso): drop docstring from the SAML proxy pair fixture
* test(sso): cover the SAML login and metadata endpoints
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): allow model instances in LiteLLM_EndUserTable validation
* fix(proxy): preserve property-backed attributes when coercing spend
* refactor(proxy): default null end-user spend with a field validator
* fix(proxy): drop model-wide from_attributes from the end-user table
* test(proxy): unskip the /customer/block integration contracts
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path
A JWT mapped to a virtual key dropped the end-user claim, so spend rows had
no end user and customer budgets, limits and region never applied. The mapped
token now carries the claim, and it takes precedence over the body user and
the x-litellm-end-user-id header, which remain the fallback when the claim is
missing or empty.
* test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim
* fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat
* fix(auth): return a generic 401 for an unknown key
The wrong-key 401 shipped the internal verification table name and the
submitted key's full SHA-256 hash to any unauthenticated caller. Both now
go to the proxy's own log and the response carries only "Invalid API key
provided."
The Admin UI detected a dead session by string-matching that message, so it
now reads the structured error type instead.
* Update ui/litellm-dashboard/src/lib/http/client.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(auth): log the missing key hash where every resolve path passes
Review catch: JWT key-mapping misses resolve through IdentityStore without
touching the main key path, so logging the hash there left those callers with
nothing but the generic 401. Log it at the store's raise site instead, which
every resolve path shares, and drop the now-duplicate log.
* test(auth): inject a key-less client instead of patching the DB fetch
The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect.
Hand IdentityStore a client whose get_data returns no row instead, so the miss is
driven through the real lookup rather than around it.
* fix(auth): log the unknown key's hash once on the auth failure line
The per-lookup warnings in get_key_object and IdentityStore logged whatever value
they were handed, which on the failure-logging path is the raw key a caller sent
(a provider key included), let a newline in ?key= forge log lines, and fired on
lookups that expect a miss. The hash now rides KeyNotFoundError onto the single
existing auth failure log line next to the requester IP, with control characters
escaped. Also drops the new comments and updates skipped tests that quoted the
old message.
* test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma
* test(integration): expect the generic wrong-key 401 in the management lifecycle tests
* fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite
* test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill
* test(auth): cover the realtime upgrade refusing an unknown key without echoing it
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): cache the two-turn guardrail replays and derive the thinking pass-through timeout
* test(integration): keep the thinking pass-through timeout as on main
The derived timeout had no failing run behind it, so this PR now only fixes the guardrail cache cells
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): give the defer_loading casts a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep tool_search items on a turn with no message or tool call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep every reasoning item on a search-only turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails
* fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses
Under drop_params, defer_loading is dropped when no tool_search tool is sent,
so the 400 it would cause upstream goes away as before. A non-bool value answers
400 without drop_params and is dropped with it. A null tool-level value falls
back to the nested function value. A request carrying a tool_search tool now
bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort
none, since chat completions rejects the tool_search type.
* fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy
Store the ordered output prefix through the last tool search item, reasoning included, so a
second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses
converter now replays tool search items and namespace only when the bridge opts in, so Noma,
the complexity router and guardrail payloads keep their shape. The tool policy guardrail and
the allowlist count replayed tools only when the request can send them.
* fix(responses bridge): forward defer_loading only beside a tool_search tool
defer_loading only means something with tool_search, and backends without it
reject the key (xAI answers 403). A true value with no tool_search tool now
answers 400 naming drop_params and is dropped under drop_params; a false value
there is a no-op and is dropped.
* fix(tool_loop): keep tool search items and namespace between rounds
* fix(responses bridge): check replayed tool search loads when a deployment adds tool_search
* fix(chat): drop litellm response fields before strict OpenAI-compatible providers
A chat reply from the Responses bridge carries provider_specific_fields on the
message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral
and Together reject unknown fields, so sending that reply back to them as
history failed with a 400 or 422. Strip them in the shared helper and use it
in those providers.
* fix(caching): keep reasoning_items and tool search items on response cache hits
A cached bridge response lost reasoning_items on a non-streaming hit and lost
both reasoning_items and provider_specific_fields (tool_search_items) on a
streamed hit, so the next turn could not replay the search or the reasoning
around it
* test(integration): cover tool search item replay on the chat to Responses bridge
Proxy, SDK, authorization and chaos cells for returning and replaying
tool_search_call and tool_search_output items and the tool call namespace
* test(integration): add response cache twins for the tool search guardrail cells
Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache.
* fix(responses bridge): forward defer_loading beside a client-executed tool_search
The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one
* test(responses bridge): pin the client tool_search turn and its replay, streamed and not
* fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search
Forwarding defer_loading beside a client tool_search made GPT stop with a
search call the chat client has to answer by hand, so a plain question that
used to get tool calls got none. Keep main's behavior of dropping it there.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Brings integration tests in line with intentional behavior changes from
#46331, #46291, #46341, #46355, #46368 and #46493, moves the Databricks
non-Claude cases off llama-3 names so they no longer download a Hugging Face
tokenizer, drops the unrealistic count_tokens tools=5 case and skips the
Mantle region-prefixed v1 base case as unrealistic input
A forwarded x-api-key or a body api_key now replaces only the key of the team
deployment, keeping its api_base, as chat completions does. A body
litellm_credential_name no longer turns team credentials off on /v1/batches.
* fix(proxy): check previous_response_id ownership on /v1/responses/compact
* test(proxy): cover the unmanaged response id escape hatches on every response id route
* test(proxy): audit compact previous_response_id ownership
Add integration cells for the native /v1/responses/compact routes covering owner,
admin, cross-tenant, unmanaged, malformed, duplicated and retained ids, provider
errors, the response cache, id-security settings, an upstream outage, a held
upstream, a worker kill and a proxy restart. Unskip the provider wire cell whose
bug this branch fixes.
* test(proxy): cover the no signing key path on every response id route
A forwarded x-api-key now counts as a caller credential, so the /rag/query
search uses it over the team deployment as at the merge base. Sorting the
JWT team claim compares entries as text, so a stray number or null no
longer turns into a 401. A Gemini store with no proxy key is a 500 again,
and a non-string vector_store_id keeps the merge base's 400 message.
* test(integration): cover keys_auth endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover hashed, path and token spellings for keys_auth contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen JWT mapping lifecycle coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover reset_spend over plaintext and hashed path spellings with reset_to 0 and 0.02
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop Final in reset loop and use drain-then-fresh-refusal in warmed-peer delete proof
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): disable model info refresh in JWT lifecycle rig
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop unused discovery constant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover cache invalidation and key paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
* test(integration): cover responses endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tighten responses contract tests per review
Isolate team-only response-id ownership with a same-user different-team key,
send every denial through the OpenAI SDK on all three aliases, make the
previous_response_id affinity check deterministic, assert the whole cancel
response, prove the Cursor stream text, -fast suffixes and include_obfuscation
strip, add SDK compact cases, restore router pre-call checks after the
affinity test, and inline the root-level responses_test_support helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for every proxy worker before routing to new deployments
Restore the cross-worker convergence wait dropped when the shared helper was
inlined, so the affinity and retrieve-routing tests cannot pass before the
alternative deployment is loaded on a multi-worker gateway
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover Responses ownership and wire contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): address responses contract review
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the provider model in the conversation passthrough body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expand responses endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): restore streamed response retrieval
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for proxy workers after deployment setup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): track readiness per proxy worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover teams endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): address review on teams contract tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drain shared upstream observations at fixture setup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove team block, member and bulk permission contracts at runtime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert email admin adds and whole membership rows for team members
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drive key update grant and revoke, Z-suffixed temp budget expiry, link LIT-9257
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close review gaps in teams endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): harden team member endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): let team admins create keys for team members behind an opt-in setting
A team admin calling POST /key/generate with a team they administer and the
user_id of another member of that team got a 403, and the dashboard only
offered the "Another User" owner option to proxy admins
Add a create_member_keys entry to team_admin_editable_team_fields. When a
proxy admin turns it on, a team admin can create a team key owned by another
member of a team they administer, through the API and through the Create Key
form. It stays off by default, so nothing changes until a proxy admin opts in
With the setting off, the 403 a team admin gets for this call now names the
setting to enable instead of saying they can only create keys for themselves
* fix(ui): order the member owner picker after the team and drop the viewer from it
For a team admin the owner picker depends on the team, so it now sits under the Team field, lists only the other members of that team, and the Team field says why it is required
* fix(proxy): keep keys a team admin creates for a member from carrying more than LLM access
A key authenticates as its owner, so with create_member_keys on a team admin could create a key owned by a proxy admin who sits in their team and act as a proxy admin with it
The member now has to hold the internal_user or internal_user_viewer role, and the key is always an llm_api key, so the team admin never holds a secret that reaches management routes. Asking for a management or read_only key for a member is refused
The unit tests now check the permission decision directly instead of replacing generate_key_helper_fn, and the saved key is checked against a real proxy in the team admin gate matrix
* style(proxy): sort the team admin field permission imports
* refactor(proxy): force the member key type on the request instead of merging a second dict
* test(proxy): drive member key creation through the key generation helper
The pure helpers were covered but the wiring in _common_key_generation_helper was not. Dropping the owner role lookup, the refusal call, or the user_id handed to common_key_access_checks left every unit test green
* fix(proxy): skip the owner role lookup when create_member_keys is off
The owner role was read before the verdict checked the setting, so with the setting off a failing user table read surfaced as a 500 instead of the 403 a team admin got before, and every such request paid an extra writer read
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test(integration): cover guardrails_policies endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): let the admin submissions list carry other rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close guardrails policies review gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover guardrail auth and policy attachments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tidy guardrail policy test helpers
* test(integration): keep plain non-admin submission refusal live
* test: expand policy attachment integration coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): accept the intended 403 for plain submission keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin multi-tag header and spend row tags for policy tag scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the duplicate pydantic import left by the main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <noreply@devin.ai>
* test(integration): cover vector_stores endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen vector store route contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): update vector store route contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop owner-update BUG skip for admin-only vector store routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector_stores): cover the managed-file fallback, owner delete and per-request retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): probe vector store worker readiness by deployment id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin vector store retry header and create routings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(integration): ruff format vector store managed file tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover realtime endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): reject duplicate multipart parts and cover the transcription_sessions alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the type ignore and check the aliased transcription response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin realtime multipart part types, replayed transcription session, undeclared session fields and budget refusal path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move the refusal path note next to the expected message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert realtime refusal status before parsing and pin the replayed transcription session whole
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): sort the realtime webrtc test imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pick the transcription mint and token with one Final each
* test(integration): pin realtime transcription mint, replay, and redeem contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(integration): ruff format the realtime webrtc test
* test(integration): drop the redundant comment on the realtime SDK calls test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): type the realtime client secret refusals table
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): tighten endpoint contract assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover chat_completions endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin per-call prompts and both n=2 chat choices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover mixed path-model auth, string guardrail rejection and typed text-completion forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the prompt-spellings cache workaround and cover Azure SDK text completions on the deployments alias
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover audio endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close audio SDK clients after each test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): avoid list mutation in audio tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): keep exact upstream call check in model resolution test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover audio translations, bracket spellings, streaming and scoped alias keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin audio metadata parse sites, exact cost and speech bug pins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop inline comment on audio wire assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen audio wire and authorization contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover agents_a2a endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin agent daily spend to the logged request day and assert cleanups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect agent daily activity newest day first
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover A2A 1.0 task aliases, agent id headers, declared access groups and per-token spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert full A2A stream, header, alias, task and spend contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert forwarded a2a-version header on A2A sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): release A2A stream gate only after a full SSE frame
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin A2A trace id, stream headers, denied alias and task methods
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin A2A card protocol version
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the whole accepted push config response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover model_management endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the exact model access refusal after a rename
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for allowlist convergence in the rename test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give the rename test lint suppression its reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close reviewer gaps on model management contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): share the blocked serving assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen authorization and configuration regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): converge spend readbacks, ignore model discovery, cover slash credential names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the dashboard credential edit body round-trip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it
Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment
* test(integration): give chaos and timing tests a harness API and gate raw signals
Chaos tests killed and suspended proxy workers with raw os.kill and psutil
calls and bounded elapsed time with fixed literals, so they raced the uvicorn
worker healthcheck and failed on slow CI boxes
OwnedProxy now offers kill_worker and replacing_worker, which return only once
a replacement answers readiness with its own pid, and suspended, which is
refused unless the proxy was launched with a lenient worker healthcheck.
timing.py bounds durations against a measured control or the injected delay.
run.py measures the proxy import time once and derives readiness deadlines and
the per-test timeout from it, writing boot-calibration.json
check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time
asserted under a numeric literal) for tests/integration outside _support, and
every call site is migrated so both report zero with no suppressions
* test(integration): steer the assistants wire burst onto both workers
The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split
* test(integration): restore product-enforced timing bounds lost in the migration
Some TQ011 migrations dropped or loosened a bound whose time is the behaviour
under test. The websocket session cap only kept its floor, so a cap closing at
300 s would have passed, and several timeouts were bounded by the injected stall
instead of the configured product value
assert_enforced_within bounds an elapsed time by the configured value plus
explicit slack, with an optional floor. The websocket cap, the MCP validation
timeout, the guardrail litellm_params timeout, the prisma command timeouts, the
baseline projection lock timeout and the key metadata statement timeout use it.
The device code refusal keeps its 15 s margin, and the langfuse retried prompt
fetch is now bounded by a cold fetch of a steady prompt as well as by the
ignored retry-after
* fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations
The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat
in a ping on the killed worker for up to that long before respawning it. It now uses
the graceful stop budget, and a worker kill returns only after the victim is reaped,
so pooled connections no longer race the dead worker's socket.
Several tests on main picked the multiprocessing resource tracker as their victim and
never killed a worker. Now that they kill a real one, presidio counts streams cut by
the death, delete_model waits for the config reload that removes a deleted model from
a replacement worker, and the wedged coordination Redis test kills before the wedge
and waits for every parked publish to fail before resuming Redis.
The redis cluster test imports the harness through the tests package so it collects
in the unit job, and the per-model fallback rig exposes the owned proxy it now needs.
* test(observability): only require S3 deliveries the killed worker could not have buffered
A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test: migrate tests/proxy_behavior into integration and unit tiers (partial)
* test(integration): run settings-dependent management tests on an owned proxy
* test(integration): assert stored values in the rewritten management rows
* test(integration): pin sibling team allocations against the org guaranteed throughput cap
* test(integration): keep Prisma clients out of the shared integration conftest import path
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(teams): let proxy admins grant team admins the right to raise their team budget
Adds a raise_max_budget entry to team_admin_editable_team_fields. With max_budget alone a team admin can still only keep or lower the team budget. Adding raise_max_budget lets them raise it, capped by the organization's max_budget when the team belongs to one, while removing the budget stays with proxy admins. The entry is rejected without max_budget, /team/info reports may_raise_max_budget to the caller, and the Admin UI nests the new checkbox under Max Budget with a tooltip and shows the team admin a hint under the budget field.
* fix(ui): shorten the raise_max_budget tooltip and link it to the docs
* fix(teams): pin the organization in the guarded team budget write
A granted raise is checked against the team's organization at read time, so the write now also requires organization_id to be unchanged. A concurrent move into a budgeted org returns 409 instead of landing an uncapped raise. Also types the new test helpers.
* test(teams): type the team row store methods and the touched race test arguments
* test(teams): annotate the last fixture and parametrize arguments in the raise_max_budget tests
* test: replace 61 live logging and otel tests with offline unit and integration coverage
* test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks
Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts.
* test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request
* test: isolate the prometheus latency test on its own deployment so counts do not depend on order
* test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only
* test: drain the redis test's logging and scope otel span checks to the test's own trace
* test: drive the periodic slack flush without a wall-clock interval
* test: scope router hook events to the test, script the db clock, split a nested comprehension
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test(proxy): audit the websocket rejection log on every live route
* test(proxy): scan the stopped proxy log for the websocket crash and drop the uvicorn wording pin
* feat(proxy): add denied_passthrough_routes deny list for custom pass-through endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): harden denied_passthrough_routes against non-admin clears and dot-segment paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for typed denied_passthrough_routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): match trailing-slash deny entries, block null metadata from dropping denies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): close bulk update, encoded ?/# and ordering gaps in passthrough deny list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check denied pass-through routes against the path the forwarder sends upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type matching_denied_passthrough_route metadata mappings
* fix(proxy): treat a `/` deny entry as denying every pass-through route
Also types the new deny-list tests and drops their get_server_root_path mock in favour of unsetting SERVER_ROOT_PATH.
* test(proxy): type the deny-list tests and drop unrelated test reformatting
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
