mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
21 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yujonglee
|
ee2d7991df
|
refactor(proxy): inject deployment warning ignores through repositories (#48592)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
zoveki
|
4786dd861b
|
fix(proxy)!: scope /health by wildcard keys and team model allowlists the way auth does (#49155)
* test(proxy): cover /health scoping for wildcard keys and team model allowlists * fix(proxy): scope /health with the model access check auth runs on a request * fix(proxy): scope /health to the model group an allowed name routes to A key alias, team alias, or router model_group_alias lets auth admit a requested name by checking its target, so /health also kept the deployment under the requested name, one the key's requests never reach. Keep only the served model group each admitted name dispatches to, through the same alias layers /v1/models mirrors, and let alias_map read its own read-only output. * test(proxy): cover /health skipping a deployment that has no model name for a restricted key * fix(health): show deployments reached through wildcard and alias allowlists, and by id when an alias shadows the name * chore(router): drop should_include_deployment from the outside-use allowlist now that /health no longer calls it * fix(health): never list a default-fallback deployment for an unserved grant, and keep access-group keys to their group's deployments * fix(health): resolve ?model= through the caller's aliases the way chat does * fix(proxy): apply user and agent model limits to /health and yield between checks /health now runs the same user and agent model checks chat runs, so a personal key whose owner may call only some models, or an agent key limited by its access group, no longer sees or probes other deployments. Each per-name check yields to the event loop, and ?model= reaches the wildcard deployment serving an allowed name * fix(proxy): check owner and agent grants for wildcard routes in /health and fail on non-access auth errors * fix(proxy): answer /health auth lookup failures the way chat does and record ?model= checks under the requested name * test(proxy): pass recorded_model_name in the direct health save tests * fix(proxy): count alias names as /health candidates so aliased names a key may call get probed * fix(health): narrow wildcard-served deployments to the key's access groups * fix(health): narrow every deployment serving a name to the key's access groups, team-owned ones included * fix(health): narrow a team deployment's internal name to the key's access groups like its public name * fix(health): keep a team deployment's internal name narrow when a limit holds only that name An agent or key limit that holds one team deployment's internal name no longer reaches every deployment behind the team's public name. The internal name maps to its public name only when the public name is also admitted. * test(health): cover all-proxy-models next to * for a key that also holds an access group * refactor(auth): keep the _as_proxy_exception name so main's tests still find it * fix(health): keep an internal team name a wildcard grant admits as its own target --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
ead221995e
|
refactor(router): move router callers onto the entry points and invert the proxy imports (#48847)
Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None. |
||
|
tavren27
|
cbf4e40d38
|
fix(proxy)!: stop labelling management and health rejections as auth failures (#48739)
* test(proxy): cover management and health routes labelling rejections as auth failures * fix(proxy): name management and health route errors by their status instead of auth_error * fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls /key/list and /key/aliases now share the catch-all helper too, so they stop adding the Authentication Error prefix and typing a 400 as internal_server_error * fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error A real outage surfaces from prisma's query engine as a builtin AttributeError, which the prisma-module check skipped, so the route catch-alls still answered 500 or 400 * fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status * fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503 * fix(proxy): answer 503 for database failovers and label provider auth errors like chat A database that refuses connections while starting up, shutting down, in recovery, or read-only after a failover now answers 503 no_db_connection instead of 400. A provider's 401 or 403 keeps the type /chat/completions gives it, while a 401 or 403 the proxy raises keeps auth_error. An exception with no message is named by its type. * fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause * test(proxy): expect the OpenAI error shape from management route refusals Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
7aa7948a47
|
fix(health): ask OpenAI and Azure audio chat models for audio output in the health check probe (#47730)
* fix(health): ask OpenAI and Azure audio chat models for audio output in the health check probe * fix(health): check the probed model when health_check_model is set, and mark Azure audio models as audio output A deployment with an audio base_model and a text health_check_model got the audio fields sent to its text probe model. The Azure gpt-audio and gpt-4o audio preview cost-map entries now set supports_audio_output, so a dated Azure name like azure/gpt-audio-1.5-2026-02-23 gets the audio probe too * fix(health): read litellm_params.base_model, and drop the stored base_model when Test Connect swaps the model * fix(health): judge the audio probe on a swapped-in health_check_params model, and keep Test Connect's stored base_model * fix(health): look up a provider-prefixed base_model under its own provider and keep text-only operator modalities |
||
|
devin-ai-integration[bot]
|
2e27a02a9f
|
feat(chatgpt): forward client subscriptions through the gateway (#41522)
* feat(chatgpt): forward each Codex user's ChatGPT OAuth login through the proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): cover _model_group_llm_provider metadata resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chatgpt): keep the gateway ChatGPT login bound to the authenticator api_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(chatgpt): move client credential header selection into the chatgpt provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(chatgpt): cover credential recognition and stored account id fallbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(chatgpt): support per-model chatgpt_auth_file for multiple ChatGPT accounts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chatgpt): resolve bare chatgpt_auth_file paths and drop redundant test docstring Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chatgpt): pass per-model auth file to early provider detection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chatgpt): normalize chatgpt_auth_file before caching the authenticator Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chatgpt): type account selection parameters * fix(chatgpt): preserve account selection through saved prompts * fix(chatgpt): allow administrator account configuration through the proxy * test(chatgpt): construct immutable request parameters * fix(router): preserve typed deployment credential resolution * fix(chatgpt): allow admins to probe unsaved account configurations * fix(chatgpt): retain SDK login headers during provider resolution * fix(chatgpt): allow admins to probe unsaved account configurations * fix(chatgpt): normalize subscription health probe input * fix(chatgpt): normalize health probe input and cover credential destinations * test(chatgpt): keep destination rejection assertions focused * feat(proxy): forward client ChatGPT subscriptions on native Codex routes * docs: lead ChatGPT forwarding with standard model setup * fix(chatgpt): preserve selected credentials in health provider lookup * fix(chatgpt): preserve gateway credentials and workspace selection * refactor(chatgpt): avoid rebinding streaming metadata inputs * fix(chatgpt): keep account lookup token argument immutable * fix(chatgpt): scope forwarded credential routing to ChatGPT * refactor: keep forwarded credential isolation opt-in by provider --------- Co-authored-by: Moe Khalil <moe@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Urbanonymous <ugutitej@gmail.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
Mateo Wang
|
9fbeebc988
|
feat(extraction): add /v1/extraction with Cloudflare Clairic support (#47473)
* feat(extraction): add Clairic extraction types and Cloudflare provider config * feat(extraction): add the /v1/extraction route, router wiring, and Clairic billing * fix(extraction): refuse or drop stream, pass unknown keys, run guardrails, keep the requested model name Clairic does not stream, so stream=true answers 400 and is dropped and billed under drop_params. Keys LiteLLM does not know pass through to Cloudflare. Guardrails scan state, field instructions and text answers. Frozen decision and extraction responses now carry the requested model name. List fields must name their item type. /health/test_connection and the Add Model page accept the extraction mode. * fix(extraction): keep upstream model names on decisions, scan images, keep typed values, refuse OpenAI params Scope the frozen-response model stamp to extraction so /v1/decisions and /v1/systemone return the upstream name as on main. Pass request images to guardrails as data URLs and drop answer spans when a response guardrail rewrites state_text. Field spec values keep their JSON type, so a wrong type answers 400 instead of being coerced. OpenAI generation params answer 400 unless drop_params. The Model Hub shows a /v1/extraction snippet for extraction models. * fix(extraction): send only Clairic's request fields, count extraction tokens, keep typed answers typed, drop spans a request guardrail made stale, and honor --max_tokens * fix(extraction): scan number answers in response guardrails, rewrite guardrail fields in linear time, and leave --max_tokens as on main * fix(extraction): scan float answers as whole numbers so fraction digits never trip a response guardrail * fix(extraction): round only mean and median for response guardrails so fractional values keep their decimal text * fix(extraction): count state and fields for budget reservations and scan estimate quantiles in response guardrails * fix(extraction): send each repeated text to guardrails once and keep the context-window budget reservation * test(extraction): type request_data in the guardrail test helpers |
||
|
Mateo Wang
|
6c4c134f23
|
fix(secret_managers)!: resolve oidc/file and oidc/env refs in admin-configured api_key per request (#47341)
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper now reads the referenced file or env var on each call and never writes the token back, so a rotated token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400 * fix(secret_managers): strip only the leading oidc/ from a secret reference A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed * fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them The advisor tool's own api_key now answers a 400 when it holds an oidc/ reference, since its sub-call runs through the client wrapper that resolves one. Stored fallback targets, auto-router tiers and user_config refuse only an oidc/ api_key, inline or through the credential they name, so their workload identity federation handling stays as it was. Provider model discovery for a wildcard deployment resolves the reference before calling the provider. * fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments * fix(utils): resolve oidc api_key refs only from admin-configured sources Mark an oidc/ api_key read from a proxy-level deployment or a stored credential, and resolve only that marked value. Team deployments, prompt templates, nested configs and a direct SDK api_key send the text as written. An unsupported reference kind raises a 401 so the router fails over to the group's next deployment. * fix(utils): send a credential's oidc token only to its own api_base host A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the credential's own api_base or base_url, so a team model or vector store that names the credential with another api_base gets a 401 and that host gets nothing. Saving a team model with an inline oidc/ api_key answers 400, since team rows never resolve one. * fix(proxy): send an admin oidc api_key only to the hosts configured beside it Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths. * fix(proxy): redact the token path from token count auth errors * test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs * fix(proxy): redact the token path from secret reference errors where they are raised * fix(proxy): keep the secret reference's path out of the error a caller sees The 401 for an unreadable oidc api_key or pass-through credential now names only the reference kind and a fixed reason, so a token path under /run/secrets or a custom credential dir never reaches the caller, while the server log keeps the full reference for the operator * fix(secret_managers): strip line breaks from the logged secret reference A reference holding a newline can no longer forge a server log line * fix(secret_managers): label an unknown oidc reference kind instead of echoing it An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in full in the caller's 401 and in the spend log, since the kind was whatever text followed oidc/. A kind get_secret does not know now reads oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>, which the client-message redaction on chat routes leaves whole. * test(health): run the real model auth check in the oidc test connection test * test(secret_managers): read an oidc/file token stored under a directory named oidc * fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references A huggingface/* wildcard call could name its own endpoint URL as the model and receive the resolved token. The origin check now treats a URL in the model as a host the reference must allow. The integration contract also reads the proxy's own database instead of the pytest worker's. * fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443, though both reach the same server. The origin check now compares scheme, host and effective port, as assert_same_origin does, and an unparseable api_base is refused instead of raising. The advisor test sets the real proxy flag instead of replacing our own helpers. * test: type the oidc request-body helpers and drive the tier test through mock_response * fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host |
||
|
Yassin Kortam
|
039dea2642
|
fix(proxy): match unnamed OTel V2 logger to otel on /health/services (#46394)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
912ad65376
|
fix(proxy): accept every known logging callback on /health/services (#46297)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
a0ac9b4f1f
|
feat(proxy): add deployment warning endpoints with instance-wide ignore and snooze (#46347)
* feat(proxy): add deployment warning endpoints with instance-wide ignore and snooze
GET /health/deployment_warnings returns the active deployment warnings
(env credential login, license expiry, no Redis on multiple workers,
detailed debug logging) with a severity and ignore state, for proxy admins
and proxy admin viewers. PATCH /health/deployment_warnings/{warning_id}
lets a proxy admin ignore a warning forever or snooze it for up to 30 days,
or restore it. Ignores live in one LiteLLM_UISettings row, apply to every
admin, and are written to the config audit log. Expired snoozes drop out
on read, and LITELLM_DISABLE_NO_REDIS_WARNING is reported as an env var
ignore. /health/readiness/details is unchanged
* fix(proxy): keep readiness Redis short-circuit ahead of the env lookup and harden ignore storage
Readiness details again answers from Redis before reading
LITELLM_DISABLE_NO_REDIS_WARNING, so proxies with Redis and a secret
manager do not fetch it on every poll. Stored ignores require timezone
aware datetimes and are read from the writer before a rewrite
* fix(proxy): keep deployment warning ignores readable in the config audit log
The audit payload keyed ignores by warning id, so the secret matcher
redacted the env_credential_login entry and the audit row lost who
snoozed it and until when. Log a list of id and ignore entries instead
|
||
|
devin-ai-integration[bot]
|
7dc7f101d9
|
feat(decisions): bill decisions and systemone from _decisions cost map fields (#45904)
* feat(decisions): bill decisions and systemone from <provider>/decisions/<model> cost map rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(decisions): mark decisions pricing rows with mode decisions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): sync generated API types for decisions mode Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): update pricing expectations in proxy tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(decisions): price classifier calls from decision rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): drop redundant evaluation handler after main merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(decisions): inject pricing row for deployed classifier Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): ship the decisions pricing lookup without cost map rows Restore both cost maps to main. Tests inject their own <provider>/decisions/<model> rows. The Add Model dropdown hides those pricing keys and offers mode decisions instead of evaluation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): expect the Decisions mode option in the Add Model form Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(decisions): bill decisions and systemone from _decisions cost map fields Replace the provider/decisions/model pricing rows with <field>_decisions fields on the model's own row. cost_per_token overlays them for the decisions, adecisions, systemone and asystemone call types and prices through generic_cost_per_token; chat, responses and batch keep the plain fields. gpt-6-luna gains its Decisions rates (input 1e-07, output 0, cached and cache-write input 1e-07, 2x above 272k tokens). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): satisfy type gate, schema sync and test_utils schema for _decisions fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(decisions): import cost_calculator module to keep basedpyright unknown-variable count flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): drop stray fireworks import from __init__ and annotate test cost rows for the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover gpt-6-luna decisions and systemone above the 272k tier Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): keep the decisions pricing block clear of main's stale azure comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(cost_calculator): drop stale azure models doc comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4c532a5713
|
feat(decisions): accept decisions as a model mode alongside evaluation (#45474)
* refactor(decisions): rename the evaluation model mode to decisions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): canonicalize the health check mode without rebinding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): replace canonical_model_mode with is_decisions_model_mode Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): preserve evaluation mode compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle from this PR Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(dashboard): format decision mode helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
9b53ee878f
|
perf(ci): isolate and shard integration suites (#44146)
* perf(ci): distribute unit cases and reuse security sweep connections * revert(ci): reject first speed trial after new failures * perf(ci): avoid repeated test discovery and parsing * perf(ci): isolate parallel security workers * revert(ci): restore measured security execution isolation * perf(tests): reuse source scans and deduplicate cache cleanup * perf(ci): restore parallel security with safe log batching * test(ci): isolate fixture state and schema comparisons * test(ci): make schema and scheduler fixtures deterministic * test(ci): drain queued logging before fixture teardown * test(ci): preserve logging delivery without idle worker restarts * test(ci): keep guardrail sinks available during parallel bursts * test(ci): isolate full proxy lifecycle from gateway imports * test(ci): preserve first failure responses and sweep evidence * test(ci): restore documentation URL settings between cases * test(ci): isolate open MCP tools in private gateways * test(ci): group private MCP cases and handle exited workers * test(ci): retain proxy exits and MCP failure details * test(ci): repair merge interactions with main Key the schema shard's full clone on artifact-name, since main removed the unit-flag input. Give each owned Redis its own log file again. Hand the realtime transcription tests main added a logging object whose dispatch handlers are awaitable, so the per-test logging drain no longer waits on a MagicMock task forever * perf(ci): shard owned integration cases and close lifecycle leaks * fix(ci): qualify shard execution and harden interrupted cleanup * fix(ci): bound worker supervision and own fault traffic * perf(ci): reuse unparametrized fixtures across bounded case groups * fix(ci): preserve boot fault allowances and complete exit records * test(ci): tolerate startup retries and retain readiness details * perf(ci): balance integration shards by recorded duration * fix(ci): load migration database helpers only for workers |
||
|
Yassin Kortam
|
fbf4af301c
|
feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange (#45158)
* feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange * fix(ui): create a credential for credential-only auth types in Add Model * fix(microsoft_365_copilot): accept max_tokens and list the chat model * fix(microsoft_365_copilot): register provider model set * fix(ui): hide other auth types' fields when credential-only types are filtered * fix(health): do not inherit stored auth settings when a connection test brings its own api_key * docs(health): restore connection-test inheritance docstrings * fix(lint): suppress justified provider boundary casts * fix(lint): satisfy M365 type-discipline gate * fix(types): eliminate type-check gate regressions * style: shorten suppression reasons so ruff format leaves them on one line * refactor(types): drop shared-file type widening unrelated to the Copilot provider * fix(health): pass caller headers as secret_fields to connection-test probes * fix(types): type connection-test probe and token counter locally * fix(lint): drop cast import from connection-test header getter * chore: remove explanatory comments flagged by review * fix(proxy): gate OAuth client credentials behind proxy admin * fix(router): skip cooldown on caller-scoped OAuth auth failures * fix(router): skip fallback cooldown on caller-scoped OAuth auth failures * fix(router): type caller-scoped cooldown lookups * fix(ci): include Microsoft 365 Copilot tests in unit shard * fix(proxy): keep saved endpoint when test_connection retests a deployment by id * fix(m365): collapse doubled Graph replies * fix(m365): preserve trailing system messages --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0ae62c5d02
|
feat(ui): add the evaluation mode to the Add Model form (#45481)
Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0ed1c08f02
|
feat(anthropic): workload identity federation and pluggable identity sources (#44448)
* feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * fix(proxy): let a deployment write name a federated credential reject_federated_credential_reference runs from is_request_body_safe, which pre_db_read_auth_checks calls on every route, so it also fired on POST /model/new, /model/update, /model/{id}/update and /health/test_connection. A proxy admin could no longer attach a federated credential to a deployment over the API or the Admin UI, leaving a static config.yaml entry as the only way to configure the feature the rejection told the caller to go configure, and _reject_non_admin_wif_write never got to make the call it exists to make. is_request_body_safe now takes the route and skips only the credential-reference check on the routes that reach can_user_make_model_call. Federation fields typed inline into a body stay refused everywhere, and a call naming a federated credential still cannot pick the identity it mints as. * refactor(proxy): derive health display policy from the federation key sets The health check module hand-copied the five workload identity fields whose value is a credential, so a shared proxy surface named provider-specific parameters and a newly added secret-bearing field would have gone on being displayed until someone remembered both places WIF_SECRET_BEARING_KEYS now sits beside the key sets it splits out of, types/utils derives secret_bearing_wif_litellm_params from it, and the health layer splats that tuple the same way it already splats the admin-only one * fix(anthropic_wif): treat blank identity-source fields as unset * test(proxy): classify the federation params in the credential slot registry main's registry test (#43298) now fails the build for any credential-named deployment param without a classification. The five federation fields that carry a token, a token file path, or a signing or client secret reference are Unplanted, matching WIF_SECRET_BEARING_KEYS; the four remaining Keycloak settings name a URL, a client id, an auth method, or a scope and are NotSecret * fix(anthropic_wif): declare federation params as owned connection leaves and chart their metrics Register the 18 Anthropic and 3 OpenAI federation params as frozen ConnectionSettings leaves so the owned-kwarg registry, the kwargs funnel and the request-body ban list read one declaration. Pass the deployment api_base through to the count-tokens handler instead of a pre-suffixed URL, which doubled the /count_tokens path on main's prompt-cache predictor. Add the five litellm_anthropic_wif_* families to the all-metrics Grafana dashboard. * fix(credentials): gate PATCH on WIF fields resolved from model_id The credential PATCH handler checked server-owned workload identity federation fields only on the values the caller sent, while a body that named a deployment through model_id had its credential values resolved after that check. A non-admin could therefore copy a federated deployment's WIF fields onto an ordinary credential. Resolve the incoming values first and run the non-admin gate on them, matching the POST path * fix(anthropic): count tokens with ANTHROPIC_AUTH_TOKEN through the shared auth header Count-tokens walked its own credential ladder: a static key, else skip minting when ANTHROPIC_AUTH_TOKEN is set, else mint a federated token. With only the auth token set it forwarded nothing and the proxy silently fell back to its local tokenizer while chat on the same deployment authenticated with that token. The handler now takes the auth header that AnthropicModelInfo.aget_auth_header resolves, the same ladder chat, files, batches and skills use, and merges the oauth beta a minted or consumer token carries with the token-counting beta --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> Co-authored-by: mateo-berri <happymvw@gmail.com> |
||
|
devin-ai-integration[bot]
|
f0eda6d2a6
|
fix(health): probe Bedrock Mantle Claude deployments over the Anthropic Messages API (#44419)
* fix(health): probe Bedrock Mantle Claude deployments over the Anthropic Messages API Bedrock Mantle serves Claude ids only on /anthropic/v1/messages, but health checks probed every chat-mode deployment over /v1/chat/completions, so a bedrock_mantle Claude deployment showed unhealthy while real /v1/messages traffic to it succeeded Add an anthropic_messages health check mode and make it the default for bedrock_mantle Claude models. An explicit model_info.mode still wins, and /health/test_connection and the Add Model form accept the new mode * fix(health): resolve the test connection mode from the deployment when the request omits it The Admin UI model page sent the mode /model/info had filled in from the cost map back as the probe mode, so Test Connection on a Bedrock Mantle Claude deployment still went over chat completions. The page now forwards only the row's id, and /health/test_connection resolves a missing mode the way /health does: the stored model_info.mode, then the mode the provider requires, then the cost map. * fix(health): resolve an omitted ahealth_check mode the way the proxy does * fix(health): test connection honors a stored mode only for the stored model and rejects a non-string mode A request that selects a stored deployment and sends a different litellm_params.model now resolves the probe mode from that model instead of the stored model_info.mode. A litellm_params.mode that is not a string answers 400 instead of 500. The Bedrock Mantle rule that Claude models are probed over the Messages API moves into the provider package. * fix(health): shape test connection probe params for the model the request probes A request that selects a stored deployment by id and overrides the model resolved its probe mode from the overridden model but still injected max_tokens from the stored mode, so an embedding override of an anthropic_messages deployment failed with a Mistral 422 extra_forbidden * fix(health): report an early ahealth_check failure as itself, not as a missing mode With the mode resolved automatically when the caller omits it, a failure before that resolution (no model, a non-string model, a provider that does not resolve) was wrapped as "Missing mode", a hint that pointed at the wrong fix and dropped raw_request_typed_dict from the result. Every failure now returns the same shape. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a76b59db9f
|
test(proxy): move middleware, spend_tracking, pass_through, common_utils and root proxy tests into tests/unit/proxy (#44015)
Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
25109a523b
|
test(proxy): move utils, agent_endpoints and endpoint tests into tests/unit/proxy (#44006)
* test(proxy): move utils, agent_endpoints and endpoint tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): package moved unit test directories Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): exclude proxy-db-owned files from the misc target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop the redundant fixture docstrings in the proxy conftest Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |