* fix(proxy): match partial wildcards against prefix-stripped model ids (issue #46384)
Refactor model checks to improve filtering logic and type hints.
* test(proxy): partial wildcard listing regression coverage (issue #46384)
* fix(proxy): drop unneeded annotation on suffix list (LIT001)
* fix(proxy): annotate suffix list with reasoned mutable-ok (basedpyright)
Change suffix_appended_wildcard_models to a typed list.
* Add lock timeout before creating telemetry report table
Set local lock timeout to 10 seconds before creating the telemetry report table.
* fix(proxy): list alias wildcards as routable ids, not raw catalog ids
An alias whose suffix collides with a catalog sub-prefix (e.g. public
"cloudflare/@cf/*" over deployment "cloudflare/*") took the partial-filter
branch and listed raw catalog ids. Routing substitutes the captured suffix
into the deployment pattern, so those listed ids routed to targets that do
not exist. Only use filter semantics when the deployment repeats the
wildcard suffix (or no deployment is available); aliases prepend their
prefix to provider-stripped catalog ids so every listed id routes back to
a real catalog model. Addresses Greptile review on #46390.
* fix(proxy): derive partial wildcard listings from the deployment's routing pattern
List an id for a partial wildcard only when routing it through the deployment's
pattern lands on a catalog model, so aliases, catch-alls, and key filters over a
wildcard deployment list ids that answer when called. /model/info expands a bare
* row the same way /v1/models does.
* fix(proxy): drop fragment ids for a catch-all over a partial wildcard and route bare aliases through the deployment pattern
A * catch-all over a partial deployment now lists no catalog ids, a bare alias like my-cf/* over cloudflare/@cf/* names ids by the deployment pattern, an alias whose wildcard count differs from its deployment lists nothing, and /model/info rows show the model each listed id routes to
* fix(proxy): keep listed ids on /model/info alias rows and clean up test-registered models
* fix(proxy): list catalog models through routing-prefix wildcard deployments
* fix(proxy): keep namespace and bare catch-all wildcards on routable listings
A leading deployment segment the model name repeats, or the provider name
itself, is no longer read as a routing segment, so same-name namespace
wildcards and a key filter over `*` -> `openai/*` stop listing ids that
route to no model. A bare `*` deployment under a prefixed name keeps the
legacy listing. `*` over a partial deployment now lists the names that
route through it, dropping empty ids.
* fix(proxy): list nothing for a catch-all over a mid-word wildcard deployment
* fix(proxy): list only wildcard ids that route back to their deployment and catalog model
A listed id now has to round-trip through the alias pattern to the catalog model it names, and the router's wildcard match for it has to pick this deployment, so an id a more specific sibling wildcard serves is no longer listed under the catch-all on /v1/models or /model/info
* fix(proxy): list wildcard aliases that route at a word boundary and ids a sibling serves with the same model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request
An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's
litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper
now reads the referenced file or env var on each call and never writes the token back, so a rotated
token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as
server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback
targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400
* fix(secret_managers): strip only the leading oidc/ from a secret reference
A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed
* fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them
The advisor tool's own api_key now answers a 400 when it holds an oidc/
reference, since its sub-call runs through the client wrapper that resolves
one. Stored fallback targets, auto-router tiers and user_config refuse only an
oidc/ api_key, inline or through the credential they name, so their workload
identity federation handling stays as it was. Provider model discovery for a
wildcard deployment resolves the reference before calling the provider.
* fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments
* fix(utils): resolve oidc api_key refs only from admin-configured sources
Mark an oidc/ api_key read from a proxy-level deployment or a stored
credential, and resolve only that marked value. Team deployments, prompt
templates, nested configs and a direct SDK api_key send the text as
written. An unsupported reference kind raises a 401 so the router fails
over to the group's next deployment.
* fix(utils): send a credential's oidc token only to its own api_base host
A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the
credential's own api_base or base_url, so a team model or vector store that names the
credential with another api_base gets a 401 and that host gets nothing. Saving a team model
with an inline oidc/ api_key answers 400, since team rows never resolve one.
* fix(proxy): send an admin oidc api_key only to the hosts configured beside it
Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths.
* fix(proxy): redact the token path from token count auth errors
* test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs
* fix(proxy): redact the token path from secret reference errors where they are raised
* fix(proxy): keep the secret reference's path out of the error a caller sees
The 401 for an unreadable oidc api_key or pass-through credential now names only
the reference kind and a fixed reason, so a token path under /run/secrets or a
custom credential dir never reaches the caller, while the server log keeps the
full reference for the operator
* fix(secret_managers): strip line breaks from the logged secret reference
A reference holding a newline can no longer forge a server log line
* fix(secret_managers): label an unknown oidc reference kind instead of echoing it
An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in
full in the caller's 401 and in the spend log, since the kind was whatever
text followed oidc/. A kind get_secret does not know now reads
oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>,
which the client-message redaction on chat routes leaves whole.
* test(health): run the real model auth check in the oidc test connection test
* test(secret_managers): read an oidc/file token stored under a directory named oidc
* fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references
A huggingface/* wildcard call could name its own endpoint URL as the model and
receive the resolved token. The origin check now treats a URL in the model as
a host the reference must allow. The integration contract also reads the
proxy's own database instead of the pytest worker's.
* fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin
A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443,
though both reach the same server. The origin check now compares scheme, host
and effective port, as assert_same_origin does, and an unparseable api_base is
refused instead of raising. The advisor test sets the real proxy flag instead of
replacing our own helpers.
* test: type the oidc request-body helpers and drive the tier test through mock_response
* fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host
* fix(proxy): refresh cached team on /team/block and /team/unblock
Both endpoints wrote `blocked` to Postgres and returned without touching the
management-object cache. Auth resolves teams through `team_id:{team_id}`
before reading the DB and `common_checks` enforces `blocked` off that cached
object, so a team whose traffic had already warmed the cache kept serving
requests after being blocked, and kept getting rejected after being
unblocked, until the entry expired.
`_refresh_cached_team` already exists for exactly this and its docstring says
every endpoint mutating `litellm_teamtable` must call it; these two were the
only mutating endpoints that didn't. The update now also asks for
`object_permission` so the refreshed cache entry doesn't drop the relation,
matching `team_model_add` and `team_model_delete`.
* fix(proxy): keep a team block from being undone by in-flight spend writes
update_cache read the cached team when a request finished and wrote that
whole copy back in a background task, so a team blocked or invalidated
while requests were in flight came back unblocked on every busy worker
until its 60 s cache entry expired. The team spend now goes onto the team
held in this worker's memory at write time, and a missing entry is left
missing for the next auth read to load.
Block and unblock responses keep litellm_model_table and object_permission
null, as before this PR, while the cache still gets both relations.
* test(proxy): type the team spend cache test parameters
* test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock
---------
Co-authored-by: Hamza Shah <hamzasepal@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Show discovered Kubernetes pods on the model page for admins
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(rag): use team provider credentials and allow viewer and JWT team access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rag): assert persisted managed store data excludes team credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(jwt): run real team model checks in the route denial test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rag): scope team provider credentials to openai vector stores
Bedrock and other providers kept their pre-PR credential and region
resolution: a team bedrock deployment's region no longer overrides the
caller's region on RAG query. Query tests now fake the provider at the
HTTP boundary with a real vector store registry.
* fix(rag): use only the team's own OpenAI deployment for RAG credentials
* fix(rag): ignore caller credential fields the query search never uses when picking team credentials
* fix(rag): treat blank credential fields as absent when picking team credentials
* test(rag): delete the stores the integration ingests create before the owned proxy exits
* fix(proxy): let a restricted key use the team deployment its wildcard or model covers
A key restricted to gpt-4o on a team whose own OpenAI deployment is openai/*
never resolved that deployment's credentials, because the key allowlist was
matched against the deployment's public name literally. The allowlist now
matches a wildcard on either side the way request authorization does, through
the auth layer's own helper, so /rag, /files and /batches resolve the team
deployment for every key that may call a model on it.
* fix(batches): read a restricted key's batch input through its team deployment
The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it
Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment
* fix(rag): load deployments oldest first, reject file_id on stores that cannot take one, and show ingest errors in the dashboard
* fix(rag): pick the team deployment with the lowest id, return 400 for caller mistakes, keep the ingest id in errors, and show partial success in the dashboard
* fix(rag): pick the team's oldest OpenAI deployment so adding one never moves the team
* fix(rag): keep config file team deployments first and try JWT teams in sorted order
A team's config file OpenAI deployments now win in file order, as on the
base, before its DB ones oldest first, so upgrading never moves a team that
has both. A JWT in several teams tries them in sorted order, so every worker
resolves the same team.
* fix(rag): return 400 for Bedrock, S3 Vectors and Gemini store config mistakes on ingest
* fix(rag): keep forwarded provider keys on /rag/query and mixed JWT team claims working
A forwarded x-api-key now counts as a caller credential, so the /rag/query
search uses it over the team deployment as at the merge base. Sorting the
JWT team claim compares entries as text, so a stray number or null no
longer turns into a 401. A Gemini store with no proxy key is a 500 again,
and a non-string vector_store_id keeps the merge base's 400 message.
* fix(rag): let a forwarded provider key win over team credentials on files, batches and ingest
* fix(rag): keep team credentials when a request carries its own key or api_base
A forwarded x-api-key or a body api_key now replaces only the key of the team
deployment, keeping its api_base, as chat completions does. A body
litellm_credential_name no longer turns team credentials off on /v1/batches.
* fix(batches): keep the team key on the team host when a request sends only api_base
A request api_base now replaces the team deployment's api_base only when the
request also brings its own api_key, and a non-string api_key or api_base is
ignored, as on the merge base.
* fix(files): ignore a blank request api_key or api_base next to team credentials
* fix(files): answer 400 for a request api_key that cannot be sent as an HTTP header
* fix(rag): honor a body drop_params for a forwarded key that cannot be an HTTP header
* fix(rag): read the request field for drop_params only on multipart ingest bodies
* fix(rag): type the ingest body read for drop_params
* fix(batches): count a model-routed batch's input with the deployment key it is created with
* fix(batches): reject a batch create whose x-litellm-model headers name different models
* fix(rag): treat a null provider and whitespace store credentials as unset when picking team credentials
* refactor(files): type the team deployment the credential resolver ranks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): take the dashboard's user role and identity from the server
Adds an internal GET /management/v1/users/me that returns the user id, role
and email the proxy resolved for the calling key. The dashboard now reads
identity from it instead of the session cookie's claims, so its navigation and
role gating always match what the backend authorizes. The layout waits on it
behind the existing loading screen, and a failed lookup clears the session and
returns to login. Sessions that must change their password can also read it
* fix(ui): sign out when the current user lookup fails and wait for it on chat
The dashboard layout now holds the loading screen and signs out when
/management/v1/users/me fails, instead of mounting the shell and looping back
to loading as each consumer refetched the failed query. The chat layout also
waits for the identity so its panels never query without a user id. The
lookup goes through the typed client with the regenerated schema.d.ts
* fix(ui): send the session's auth header on the first user lookup and keep chat closed until authorized
The first lookup runs before AuthProvider registers the custom header name, so it now reads auth_header_name from the session cookie. Chat also waits for isAuthorized, so a failed lookup no longer mounts the shell with an empty user id
* fix(ui): type the dashboard callers of the now typed user id
useAuthorized used to hand back the cookie's user id untyped, which hid six strict type errors in its callers. The id is now string | null, so the production build failed. Callers handle the null case, and query key filters accept the boolean blocked filter
* fix(ui): look up the current user only after the proxy URL is discovered
Pages outside the dashboard shell, like chat and connect, could start the lookup before UI config discovery set the proxy base URL. On a split-origin deployment both attempts then hit the UI origin and a valid session was signed out
* fix(ui): keep a loaded identity while UI config discovery reruns
Pausing the lookup by passing a null token switched the query key, so on the dashboard the identity the layout had already loaded vanished until the shell's own UI config request finished. The key now stays the same and only the fetch waits for discovery
* feat(anthropic): workload identity federation and pluggable identity sources
Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one
commit on top of litellm_internal_staging without the dashboard changes.
Deployments on anthropic/ without a static api_key can exchange an OIDC
workload assertion for a short-lived sk-ant-oat01 token through a shared
RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file,
an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment,
per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation
fields are server-owned: refused inline in request bodies and on
POST /model/new, proxy-admin only on credentials, and the token exchange
is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS
adds a host. GET /credentials/{name}/jwks exports the public key set of a
LiteLLM-signed credential for the Claude Console.
The OpenAI federation trio from #39613 rides along on the backend side with
the same server-owned handling.
Fixes#28607
Resolves LIT-6107
Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
* fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries
The files handler enabled workload identity on batch-result downloads but never received the
deployment's litellm_params, so a deployment authenticating through a named credential could only
mint from process-wide env vars. It now threads litellm_params through to the auth header the way
the batch retrieve path already does.
LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme,
so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the
gateway. Entries are now parsed as network locations whether or not they carry a scheme.
* fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle
* test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries
* fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly
* fix(proxy): decrypt stored litellm_params before the WIF write gate
* fix(proxy): hide WIF secret references from /health output
* fix(proxy): keep the proxy error shape on credential endpoint refusals
* fix(proxy): hide identity token file paths from /health output
* feat(openai): honor per-deployment workload identity on non-chat routes
Image generation, audio speech and transcription, moderation, text
completions, files, batches, and fine-tuning built their OpenAI client from
the env vars alone, so a keyless deployment carrying
openai_identity_provider_id, openai_service_account_id, and
openai_identity_token_file failed with the api_key error on every route
other than chat, Responses, and embeddings. Those handlers now build their
client through the shared workload identity resolver, and the managed batch
file-access credential extraction keeps the trio
* fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys
Non-chat handlers built a fresh SDK client per call, so files, batches,
fine-tuning, text completion and sync moderation exchanged the OIDC token on
every request. The builders now reuse the cached workload identity client per
identity, base, timeout, retries and organization; static-key clients are
still built fresh.
A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no
longer silently overrides a deployment's openai_identity_* params. An explicit
distinct api_key on the deployment or call still wins.
* fix(openai): keep shared workload identity clients on the process http session
The workload identity client is cached and shared across requests, so it must
never borrow the transport a single caller passed in. Static-key clients keep
using the caller's http client; federated clients always ride the process-wide
session. Also makes the http client helpers on BaseOpenAILLM public since the
builder now needs them from outside the class.
* test(openai): pin partial deployment identity beating the process key
* test(openai): type the managed-object invoke callback
* fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working
The Bedrock Claude Platform route already reads anthropic_workspace_id from
optional_params, so banning that spelling as a server-owned federation
parameter broke a pre-existing client capability. The federation field is now
anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID),
which restores the base branch's behavior for Bedrock callers, drops the
Bedrock-specific hint from the refusal message, and deletes the unconditional
ban constant that no longer had a reader
* fix(auth): share one exchanged token across workers reading the same assertion
Anthropic accepts each identity assertion exactly once, so two uvicorn
workers reading the same token file both minting from it means the second
exchange is denied with jti_reused. Minted tokens now land in a per-user
0700 cache directory guarded by a file lock, so workers on the same host
reuse one exchange until the token expires or the assertion rotates. A 401
is only retried when the re-read assertion actually differs, and the denial
hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache
and an empty value disables it
* feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals
The httpx-based OpenAI handlers built their Authorization header from the
deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying
openai_identity_provider_id, openai_service_account_id, and
openai_identity_token_file sent "Bearer None". They now resolve the bearer
through resolve_openai_bearer_token, which exchanges the deployment's
identity token the same way the SDK-based routes do.
A deployment that carries only part of the identity trio now raises a 500
naming the missing field instead of silently falling back to the process
key, and the SDK client builders take a factory for the static-key http
client so the federated branch never touches litellm.client_session.
* fix: keep anthropic federation from being shadowed or leaked
An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated
deployment sent an empty x-api-key on every call instead of minting a token.
Blank values now read as unset, and a real static key on a federated deployment
logs once that it outranks federation and nothing is being federated.
The exchange-host allowlist matched hostnames only, so a second process on
another port of an allowed host was trusted with the workload's identity token.
An entry that names a port now trusts that port alone, while a bare host still
trusts every port.
The shared token store exists so the workers reading one projected token file do
not each spend its single-use jti. A source that mints its own assertion per
exchange shares nothing with another worker, so it no longer writes a live token
to disk for a lookup that can never hit.
* feat(openai): resolve per-deployment workload identity for the assistants routes
OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders.
* fix: unlink a staged token file a failed write leaves behind
The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads.
* refactor: move anthropic jwks derivation behind a provider-owned tagged union
* feat(openai): resolve per-deployment workload identity for image variations
Build the image variations client through the shared workload identity
builders instead of scraping locals(), and give the async path the async
http session it was missing. Let a federation trio serve the model-less
batch fallback, and hold the standard LRU size of exchanged tokens.
* fix: unlink the staged token file when its write fails at close
A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token.
* fix(anthropic): close the staging descriptor before writing the shared token file
* fix(wif): judge federation writes by what they set, not what is stored
The admin gate read the stored deployment, so a team admin lost edit, delete
and Test Connection on any deployment carrying federation params. It now
returns early unless the submitted fields touch the federation surface, and a
Test Connection probe that points the deployment at its own api_base is still
refused, with the 403 no longer wrapped into a 500
The rest of the same review pass: POST /model/new refuses only a blocking
value of `blocked`, so a client that always sends `blocked: false` is not
turned away; a request body can no longer pick which federated identity to
mint as by naming a stored credential; an advisory refresh the executor
refuses disarms the entry instead of wedging the identity until the follower
timeout; the static-key shadow warning resolves its env fallback inside the
cache instead of once per request; credential writes drop nulls before
storing them; the token exchange validates the endpoint URL before reading an
assertion and keeps refusing redirects across a client heal; /health hides
every server-owned federation field from non-admins; and the async create_file
and create_batch paths say which setting is missing when the provider resolves
no URL
* style(tests): keep two blank lines before the batch_cost_is_final block
* test(batches): type the workload identity fallback test fixtures
* fix(moderation): stop validating unrelated kwargs on the sync workload identity path
* fix(auth): refuse server-owned workload identity keys in array form and inside a client router config
* fix(openai): keep a deployment's own key ahead of workload identity on every route
Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation,
image generation, and assistants so a deployment key equal to the process key still wins. A partial
identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and
/v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments
* fix(containers): pass deployment params to container file auth headers
Container file routes built their auth headers without the deployment's
litellm_params, so a keyless deployment with an OpenAI workload identity
sent Bearer None on upload, list, retrieve, delete and content calls
* fix(openai): keep process-wide key precedence and move token exchange off the event loop
Restore the merge base order where an explicit or process-wide OpenAI key wins over
federation, so exporting OPENAI_API_KEY keeps working the way it did before.
Run the header-building validate_environment calls (image edit, videos, containers,
vector stores, vector store files) and the decisions and System One request
preparation in a worker thread, so a slow token exchange no longer stalls the loop.
Federate realtime websocket sessions, live sessions, realtime client secrets and
transcription sessions, and the OpenAI moderation guardrail when no key is set.
* fix(openai): answer 401 for a partial identity trio and federate realtime health checks
A deployment with part of the workload identity trio and no static key now
fails as an auth error naming the missing params, so the router does not retry
it as a server error. The realtime health check sends the exchanged bearer for
a keyless federated OpenAI deployment, and API token counting reads the same
static keys as every other route before minting.
* fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config
* fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting
* test(integration): refuse caller-chosen identity params in user_config and bracketed form keys
* fix(openai): send the honored secret manager key from SDK client builders
A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key
on image, assistants, files, batches, and other SDK-built routes instead of failing with
no api_key, matching what the workload identity resolver already treats as a static key
* test(openai): check the token exchange runs off the event loop instead of timing the loop
---------
Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path
A JWT mapped to a virtual key dropped the end-user claim, so spend rows had
no end user and customer budgets, limits and region never applied. The mapped
token now carries the claim, and it takes precedence over the body user and
the x-litellm-end-user-id header, which remain the fallback when the claim is
missing or empty.
* test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim
* fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat
* fix(auth): return a generic 401 for an unknown key
The wrong-key 401 shipped the internal verification table name and the
submitted key's full SHA-256 hash to any unauthenticated caller. Both now
go to the proxy's own log and the response carries only "Invalid API key
provided."
The Admin UI detected a dead session by string-matching that message, so it
now reads the structured error type instead.
* Update ui/litellm-dashboard/src/lib/http/client.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(auth): log the missing key hash where every resolve path passes
Review catch: JWT key-mapping misses resolve through IdentityStore without
touching the main key path, so logging the hash there left those callers with
nothing but the generic 401. Log it at the store's raise site instead, which
every resolve path shares, and drop the now-duplicate log.
* test(auth): inject a key-less client instead of patching the DB fetch
The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect.
Hand IdentityStore a client whose get_data returns no row instead, so the miss is
driven through the real lookup rather than around it.
* fix(auth): log the unknown key's hash once on the auth failure line
The per-lookup warnings in get_key_object and IdentityStore logged whatever value
they were handed, which on the failure-logging path is the raw key a caller sent
(a provider key included), let a newline in ?key= forge log lines, and fired on
lookups that expect a miss. The hash now rides KeyNotFoundError onto the single
existing auth failure log line next to the requester IP, with control characters
escaped. Also drops the new comments and updates skipped tests that quoted the
old message.
* test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma
* test(integration): expect the generic wrong-key 401 in the management lifecycle tests
* fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite
* test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill
* test(auth): cover the realtime upgrade refusing an unknown key without echoing it
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(telemetry): count Admin UI page views and tab clicks through the proxy
* fix(ui): share one telemetry settings save across both tabs so a second toggle waits for the first
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): let every dashboard role reach the UI event routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): fail with the denial detail when a dashboard role cannot reach UI event routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): document POST /telemetry/ui_events with a description and a 200 response schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep the test-quality reason on the line pytest's checker reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): pick the telemetry destination on the Telemetry page and warn when none is chosen
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): send the destination along with groups in the telemetry settings hook test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add a top-level telemetry switch and gate turning telemetry on behind an enterprise modal
The page can only turn telemetry off. Turning it on from the switch, a group
or a first destination opens a modal: enterprise proxies are told to contact
their LiteLLM enterprise representative, others that telemetry is an
enterprise-only feature. Group switches and the destination stay visible but
disabled while telemetry is off.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the personal budget on a proxy-admin user object
A token carrying the proxy_admin role (JWT admin scope, SSO) replaced the fetched LiteLLM_UserTable row with a bare admin object, so max_budget never reached common_checks and admins spent past their personal budget on /chat/completions. Keep the row and force only user_role.
Co-authored-by: L4XB <lukas.buck@e-mail.de>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): budget denial for a proxy-admin JWT is the 422 ExceededBudget response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): carry only the personal budget onto a forced proxy-admin user object
Keeping the whole DB row also carried its models list, so a proxy admin
whose row had no-default-models or a restricted list got 403 on personal
requests. Build the admin object as before and copy max_budget and
user_email across. The built-in admin identity (the master key and the UI
username/password login) keeps ignoring its row's budget.
* test(proxy): pin the contrast case in the admin model-list and admin-name budget tests
Each test now also shows the same cached row refusing a non-exempt token, so the
pass case cannot go green on a row the gate never read.
* test(e2e): drive the personal-budget case with a proxy_admin user row and a plain JWT
The Buildkite stack keeps the default admin_allowed_routes, so an admin-scope JWT
gets 403 on /chat/completions before any budget check and the case could never
pass there. A plain token whose sub resolves to a proxy_admin row goes through the
same forced-admin rebuild. Drop the stage-mirror admin_allowed_routes edit, which
the deployed stack never read.
* ci: rerun e2e after an unrelated Gemini streaming spend-row timeout
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: L4XB <lukas.buck@e-mail.de>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test: move guardrails, image, audio, batches, search legacy tests to unit
* test: restore the custom guardrail and remaining chunk 1 unit rows
* test: move the ocr matrix to e2e
* test: drop comments from moved tests and fix vertex ocr credentials
* test: move litellm utils and unified google legacy tests to their tiers
* test: move logging callback and pass-through unit legacy tests to their tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move store_model_in_db legacy tests and start the admin ui migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: finish the proxy job rows and drop their CircleCI jobs
Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI
* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names
* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist
* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in
* test: type the e2e beta header, prompt cache and secret manager values
* test: assert the missing model error message on the OpenAI routes
* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks
* test: make the langfuse fixture directory a package for the layout check
* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane
* test: keep the ocr e2e pdf fixture next to the test
* test: put meta below covers on the streaming tool search e2e
* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures
* test: count only OCR calls in the OCR matrix recorder
* test: pin OCR parse bodies and billed cost, group new e2e cases into classes
* ci: restore the python image anchor dropped in the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy/auth): expand all-team-models sentinel for project model access checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): use CI-served models and tighten project all-team-models denial check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): declare coverage registry cells for project all-team-models tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger e2e after ephemeral stack deploy timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): wait for team allowlist propagation before project denial check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): annotate written_at as Final per LIT010
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jesus <jesus@berri.ai>
* feat(proxy): require trace IDs per team on LLM, MCP and agent requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert trace ID authorization outcomes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear trace ID type gate errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format trace ID type check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): validate trace ID metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use read-only trace metadata types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip trace ID metadata validation for teams without dict metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): check team trace ID setting before route matching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): sign RDS IAM tokens for the database's own region
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add per-connection RDS IAM signing region overrides for writer and reader
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): integration cells for per-connection RDS IAM signing regions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): opt-in real RDS e2e cells for cross-region IAM signing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): address review on RDS IAM region cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop recording-front integration tests, mark RDS e2e suite e2e and register its model via /model/new
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert the spend read lands on the replica, not just a live connection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type generate_iam_auth_token params and retry the replica spend-read check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): derive RDS region from custom RDS Proxy endpoint hostnames
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): add Subject, step labels, frozen rows and Final to the RDS IAM e2e suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop the opt-in RDS IAM e2e suite in favor of unit coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
* feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange
* fix(ui): create a credential for credential-only auth types in Add Model
* fix(microsoft_365_copilot): accept max_tokens and list the chat model
* fix(microsoft_365_copilot): register provider model set
* fix(ui): hide other auth types' fields when credential-only types are filtered
* fix(health): do not inherit stored auth settings when a connection test brings its own api_key
* docs(health): restore connection-test inheritance docstrings
* fix(lint): suppress justified provider boundary casts
* fix(lint): satisfy M365 type-discipline gate
* fix(types): eliminate type-check gate regressions
* style: shorten suppression reasons so ruff format leaves them on one line
* refactor(types): drop shared-file type widening unrelated to the Copilot provider
* fix(health): pass caller headers as secret_fields to connection-test probes
* fix(types): type connection-test probe and token counter locally
* fix(lint): drop cast import from connection-test header getter
* chore: remove explanatory comments flagged by review
* fix(proxy): gate OAuth client credentials behind proxy admin
* fix(router): skip cooldown on caller-scoped OAuth auth failures
* fix(router): skip fallback cooldown on caller-scoped OAuth auth failures
* fix(router): type caller-scoped cooldown lookups
* fix(ci): include Microsoft 365 Copilot tests in unit shard
* fix(proxy): keep saved endpoint when test_connection retests a deployment by id
* fix(m365): collapse doubled Graph replies
* fix(m365): preserve trailing system messages
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on
* fix(bedrock): sign the listing query RFC 3986 style and raise BedrockError on a failed listing
* test(utils): keep test_utils.py at main's formatting, only the Bedrock discovery test is new
* fix(bedrock): chain BedrockModelInfo's constructor so the passthrough config keeps its event parser
* fix(bedrock): list discovered ids without the provider prefix so partial wildcards filter
get_valid_models returns bare vendor ids for every other provider and for the static bedrock catalog, and the proxy's wildcard expansion adds the provider prefix itself. The lister prefixed its ids, so a partial wildcard like bedrock/anthropic.* never matched the proxy's filter and listed uncallable bedrock/anthropic.bedrock/<id> entries
* test(bedrock): integration cells for wildcard discovery through the proxy and the SDK
* fix(proxy): keep a partial wildcard a filter when its deployment repeats the prefix
The wildcard expansion guessed filter-or-alias by whether any provider id carried the prefix. A deployment such as model_name bedrock/anthropic.* over model bedrock/anthropic.* can only ever route names that share the prefix, so when the account has no on-demand anthropic.* id (every Anthropic model behind an inference profile) the guess fell into the alias branch and listed bedrock/anthropic.<every id>, none of them callable. A deployment whose model repeats the suffix now always filters, and lists nothing when nothing matches
* fix(proxy): prefix every discovered id under a custom wildcard prefix that starts a vendor id
* test(bedrock): Final and read-only annotations in the wildcard discovery tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit
Relocate 53 legacy tests that pass with no network or keys into the tests/unit
files that mirror the code they exercise, drop 2 llm_guard tests already covered
by test_llm_guard_call_type_aliases, and delete the emptied legacy files.
* test: freeze the reset clock and run the real reset methods in the moved budget reset tests
Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await
the service-hook tasks the job spawns instead of sleeping. Drive the per-user
failure with a malformed budget_duration row from the prisma mock so the real
_reset_budget_for_key/user/team run instead of patched replacements.
* test: use fixed timestamps in the moved pass-through and spend log tests
* test: mock the moderation HTTP call and type the moved moderation and spend log tests
Serve the OpenAI moderation response through an httpx MockTransport so the real
async_make_request runs, annotate the moved tests' locals with Final, and pass
get_logging_payload typed kwargs instead of an untyped input_args dict.
* test: drive the moved proxy tests through boundaries and annotate their locals with Final
Serve GCS downloads and the Router moderation call through httpx mocks with fake
Google credentials, let the real pass-through success handler fail on the mocked
upstream response, and fold the Vertex live route endpoint check into the existing
route test. Annotate moved locals with Final, build the budget reset rows without
rebinding, and type the remaining fakes without **kwargs
* feat(fireworks_ai): forward the LiteLLM user id as user behind fireworks_forward_user_id
* fix(fireworks_ai): keep the forwarded user id when the caller sends extra_body.user
* fix(proxy): answer a rejected websocket handshake without crashing the HTTP exception handler
* fix(proxy): read the OTLP route check's scope keys with .get and drive the websocket auth tests through the real auth path
* test(proxy): build the websocket auth test keys without reading the clock
* fix(proxy): admit litellm_proxy/hosted_vllm in provider-endpoint discovery
get_provider_models gated all endpoint discovery on membership in the
static litellm.models_by_provider dict, which litellm_proxy and
hosted_vllm are intentionally absent from since their model list only
exists behind the provider's own endpoint. GET /v1/models returned the
literal wildcard string instead of the expanded model list for these
providers. Now falls through to ProviderConfigManager, which already
knows about them, before giving up.
* fix(tests): stop patching an SDK internal in the discovery regression test
TQ008 flagged patching litellm.proxy.auth.model_checks.get_valid_models.
Assert the gate's own return value instead of mocking past it.
* chore: drop redundant comment per repo convention
* test(proxy): cover litellm_proxy wildcard model discovery end to end
---------
Signed-off-by: mayuriphad <163738104+mayuriphad@users.noreply.github.com>
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add denied_passthrough_routes deny list for custom pass-through endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): harden denied_passthrough_routes against non-admin clears and dot-segment paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for typed denied_passthrough_routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): match trailing-slash deny entries, block null metadata from dropping denies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): close bulk update, encoded ?/# and ordering gaps in passthrough deny list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check denied pass-through routes against the path the forwarder sends upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type matching_denied_passthrough_route metadata mappings
* fix(proxy): treat a `/` deny entry as denying every pass-through route
Also types the new deny-list tests and drops their get_server_root_path mock in favour of unsetting SERVER_ROOT_PATH.
* test(proxy): type the deny-list tests and drop unrelated test reformatting
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
* test: move the unit half of 126 mixed legacy files into tests/unit
* test: restore litellm globals that moved tests set
* test: finalize migration test cleanup
* test: restore original bodies of moved legacy tests
The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit
test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated
* test: drop module imports shadowed by restored local imports
* test: assert on LiteLLM output in no-assertion moved tests and isolate leaks
Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change
Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags
* test: drain queued logging tasks before the Phoenix span test
The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(proxy): add opt-in vector_store_deny_by_default for standalone virtual keys
Adds general_settings.vector_store_deny_by_default (typed bool, default false). When enabled, a virtual key
with no team must list the requested vector store in its object_permission.vector_stores; no permission
record, null, or an empty list is denied with key_vector_store_access_denied. Omitted or false keeps the
existing behavior, including nonempty allowlist enforcement. The master key is unchanged in both modes.
Team keys and keyless callers are deferred to later increments of LIT-6035
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): require key and team vector store grants for team keys under vector_store_deny_by_default
With the flag enabled, a virtual key on a team needs both its own grant and its team's grant for every requested vector store. A missing permission record, an empty list or an unresolved team grants nothing. Dashboard session keys and the master key keep their existing behavior, and flag-off behavior is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): require user or team vector store grants for keyless requests under vector_store_deny_by_default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover vector_store_deny_by_default through a real proxy with key, team and JWT identities
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for vector_store_deny_by_default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): cover path and file_search vector store ids under vector_store_deny_by_default
Strict mode now reads vector_store_ids and tools[].vector_store_ids from the request body without needing a vector store registry, so /v1/vector_stores/{id}/search and Responses file_search are checked. User grants load through the object permission cache, and the proxy admin user rebuild keeps object_permission_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast user entitlement cache eviction to every worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): reuse VectorStoreRegistry id extraction for vector_store_deny_by_default
Strict mode now uses get_vector_store_ids_to_run on an empty registry when none is loaded, instead of a parallel set of request-shape helpers, and vector_store_access_check documents the policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add strict vector store audit cells for routes, SDKs, workers and concurrency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): consolidate vector_store_deny_by_default coverage to core cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject invalid vector_store_deny_by_default at config load and return 400 for malformed vector_store_ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): read vector store key and team grants through the object permission cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): return typed permission rows in request flow vector store tests
Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): validate vector store ids and tools as immutable sequences
Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): replace Any with proven types in 299 files
Clears 871 basedpyright Any errors (reportAny 6,703 to 6,240, reportExplicitAny 1,651 to 1,243) without adding a cast, an ignore or a suppression, and without touching any budget file
Most edits are annotation-only: a parameter, return or local goes from Any to object, Mapping[str, object] or the concrete type the value always held. Fourteen files validate untyped JSON once where it enters, through a module-level pydantic TypeAdapter or model_validate, and then use real types
No HTTP status, error type or response shape changes. Mistral speech and fal.ai Bria image generation now report a pydantic ValidationError instead of an AttributeError when the provider answers 2xx with a body that is not a JSON object
* refactor(types): make the config locals fix effective and trim no-op edits
The Mapping[str, object] annotation on `locals().copy()` removed no error,
because the checker narrows the variable back to the dict[str, Any] the call
returns. 21 provider config constructors now build the same copy with
dict(locals()), which the checker infers as object values under that
annotation, so each file loses one reportAny.
The same annotation is reverted in 31 other config files where it stayed a
no-op, together with the tests that were added only to cover those lines, and
the one MCP server manager line that no CI coverage shard executes is
reverted too. The pull request drops from 345 to 297 changed files.
* refactor(types): accept only int in the proxy state setter
get_proxy_state_variable is annotated to return int, but
set_proxy_state_variable still took Any, so the checker could not hold
callers to the type the getter promises. The setter now takes int, which is
what its only caller already passes.
* refactor(types): index the proxy state key so the getter returns int
* refactor(types): keep public annotations and provider error text unchanged
Restore every public return, public method parameter, public attribute and exported
alias to its annotation on main so code that type-checks against the package keeps
type-checking, and take the Mistral speech and fal.ai Bria changes back out so no
provider error message differs from main
* refactor(types): leave the Vertex RAG chunking read as it is on main
Take the chunking format validation back out of the Vertex RAG ingestion path. It needs the vertexai SDK, a storage bucket and a RAG corpus to execute, so nothing here could run it end to end, and it cleared only two errors
* test(integration): pin the validated provider boundaries on a live proxy
* test(integration): give the held burst a client that outlasts the gate
The fault cell holds a burst at the upstream for up to 60 seconds while it kills a worker, but sent the burst through the shared 15 second client, so a slow box could time the survivors out before the gate opened. The burst now goes through its own client whose timeout is twice the gate, and the gate length is one named constant.
* refactor(types): keep the license reply handling and experimental MCP signatures as they were
The license check validated the whole reply as a mapping, which changed the error text logged for a reply that is not an object. It now validates only the verify value, so every reply is handled and logged exactly as before while the value is still typed.
Three files under the experimental MCP server changed annotations on public functions and methods (three returns and three parameters). They go back to their previous content so no public signature in the diff is narrowed.
* test(integration): answer the proxy's model-list call in the OpenAI stand-ins
Every 300 seconds each proxy worker asks an OpenAI deployment for GET /v1/models. Four new cells own an OpenAI stand-in that accepted only the call under test, so a refresh landing inside a cell failed it. The stand-ins now answer that call through the suite's own helper and the cells count only the provider calls they drive.
