mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-19 04:54:37 +00:00
b39eadcb82
* feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * feat(openai): honor per-deployment workload identity on non-chat routes Image generation, audio speech and transcription, moderation, text completions, files, batches, and fine-tuning built their OpenAI client from the env vars alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file failed with the api_key error on every route other than chat, Responses, and embeddings. Those handlers now build their client through the shared workload identity resolver, and the managed batch file-access credential extraction keeps the trio * fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys Non-chat handlers built a fresh SDK client per call, so files, batches, fine-tuning, text completion and sync moderation exchanged the OIDC token on every request. The builders now reuse the cached workload identity client per identity, base, timeout, retries and organization; static-key clients are still built fresh. A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no longer silently overrides a deployment's openai_identity_* params. An explicit distinct api_key on the deployment or call still wins. * fix(openai): keep shared workload identity clients on the process http session The workload identity client is cached and shared across requests, so it must never borrow the transport a single caller passed in. Static-key clients keep using the caller's http client; federated clients always ride the process-wide session. Also makes the http client helpers on BaseOpenAILLM public since the builder now needs them from outside the class. * test(openai): pin partial deployment identity beating the process key * test(openai): type the managed-object invoke callback * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals The httpx-based OpenAI handlers built their Authorization header from the deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file sent "Bearer None". They now resolve the bearer through resolve_openai_bearer_token, which exchanges the deployment's identity token the same way the SDK-based routes do. A deployment that carries only part of the identity trio now raises a 500 naming the missing field instead of silently falling back to the process key, and the SDK client builders take a factory for the static-key http client so the federated branch never touches litellm.client_session. * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * feat(openai): resolve per-deployment workload identity for the assistants routes OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * feat(openai): resolve per-deployment workload identity for image variations Build the image variations client through the shared workload identity builders instead of scraping locals(), and give the async path the async http session it was missing. Let a federation trio serve the model-less batch fallback, and hold the standard LRU size of exchanged tokens. * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * style(tests): keep two blank lines before the batch_cost_is_final block * test(batches): type the workload identity fallback test fixtures * fix(moderation): stop validating unrelated kwargs on the sync workload identity path * fix(auth): refuse server-owned workload identity keys in array form and inside a client router config * fix(openai): keep a deployment's own key ahead of workload identity on every route Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation, image generation, and assistants so a deployment key equal to the process key still wins. A partial identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and /v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments * fix(containers): pass deployment params to container file auth headers Container file routes built their auth headers without the deployment's litellm_params, so a keyless deployment with an OpenAI workload identity sent Bearer None on upload, list, retrieve, delete and content calls * fix(openai): keep process-wide key precedence and move token exchange off the event loop Restore the merge base order where an explicit or process-wide OpenAI key wins over federation, so exporting OPENAI_API_KEY keeps working the way it did before. Run the header-building validate_environment calls (image edit, videos, containers, vector stores, vector store files) and the decisions and System One request preparation in a worker thread, so a slow token exchange no longer stalls the loop. Federate realtime websocket sessions, live sessions, realtime client secrets and transcription sessions, and the OpenAI moderation guardrail when no key is set. * fix(openai): answer 401 for a partial identity trio and federate realtime health checks A deployment with part of the workload identity trio and no static key now fails as an auth error naming the missing params, so the router does not retry it as a server error. The realtime health check sends the exchanged bearer for a keyless federated OpenAI deployment, and API token counting reads the same static keys as every other route before minting. * fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config * fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting * test(integration): refuse caller-chosen identity params in user_config and bracketed form keys * fix(openai): send the honored secret manager key from SDK client builders A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key on image, assistants, files, batches, and other SDK-built routes instead of failing with no api_key, matching what the workload identity resolver already treats as a static key * test(openai): check the token exchange runs off the event loop instead of timing the loop --------- Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| a2a | ||
| ai21/chat | ||
| aiml | ||
| aiohttp_openai/chat | ||
| amazon_nova | ||
| anthropic | ||
| apiserpent | ||
| aws_polly | ||
| azure | ||
| azure_ai | ||
| base_llm | ||
| baseten | ||
| bedrock | ||
| bedrock_mantle | ||
| bespoke/decisions | ||
| black_forest_labs | ||
| brave/search | ||
| bytez | ||
| cerebras | ||
| chatgpt | ||
| clarifai/chat | ||
| claude_code | ||
| cloudflare | ||
| codestral/completion | ||
| codex | ||
| cohere | ||
| cometapi | ||
| compactifai | ||
| custom_httpx | ||
| dashscope | ||
| databricks | ||
| dataforseo/search | ||
| datarobot/chat | ||
| deepagents | ||
| deepgram | ||
| deepinfra | ||
| deepseek | ||
| deprecated_providers | ||
| docker_model_runner/chat | ||
| duckduckgo/search | ||
| e2b | ||
| edenai | ||
| elevenlabs | ||
| empower/chat | ||
| exa_ai/search | ||
| fal_ai | ||
| fastcrw | ||
| featherless_ai/chat | ||
| firecrawl | ||
| fireworks_ai | ||
| friendliai/chat | ||
| galadriel/chat | ||
| gdc | ||
| gemini | ||
| gigachat | ||
| github/chat | ||
| github_copilot | ||
| google_pse/search | ||
| gradient_ai/chat | ||
| groq | ||
| heroku/chat | ||
| hosted_vllm | ||
| huggingface | ||
| hyperbolic | ||
| inception | ||
| infinity | ||
| jina_ai | ||
| lambda_ai | ||
| langflow | ||
| langgraph | ||
| laya | ||
| lemonade | ||
| linkup | ||
| litellm_proxy | ||
| llamafile/chat | ||
| lm_studio | ||
| manus | ||
| meta/realtime | ||
| meta_llama/chat | ||
| microsoft_365_copilot | ||
| milvus/vector_stores | ||
| minimax | ||
| mistral | ||
| modelscope | ||
| mongodb | ||
| moonshot/chat | ||
| morph | ||
| nadir/chat | ||
| nebius | ||
| nimble | ||
| nlp_cloud | ||
| novita/chat | ||
| nscale/chat | ||
| nvidia_nim | ||
| nvidia_riva | ||
| oci | ||
| ollama | ||
| oobabooga | ||
| openai | ||
| openai_like | ||
| opencode | ||
| openrouter | ||
| opensandbox | ||
| ovhcloud | ||
| parallel_ai/search | ||
| pass_through | ||
| perplexity | ||
| petals | ||
| pg_vector/vector_stores | ||
| predibase | ||
| ragflow | ||
| recraft | ||
| reducto | ||
| replicate | ||
| runwayml | ||
| s3_vectors | ||
| sagemaker | ||
| sail | ||
| sambanova | ||
| sap | ||
| scaledown | ||
| scaleway | ||
| searchapi | ||
| searxng | ||
| serper/search | ||
| snowflake | ||
| soniox | ||
| stability | ||
| strands_decider/decisions | ||
| tavily/search | ||
| tencent | ||
| tinyfish/search | ||
| together_ai | ||
| tool_loop | ||
| topaz | ||
| triton | ||
| typesafe/decisions | ||
| v0 | ||
| valkey | ||
| vercel_ai_gateway | ||
| vertex_ai | ||
| vllm | ||
| volcengine | ||
| voyage | ||
| wandb | ||
| watsonx | ||
| xai | ||
| xinference/image_generation | ||
| you_com | ||
| zai | ||
| __init__.py | ||
| base.py | ||
| compaction.py | ||
| custom_llm.py | ||
| maritalk.py | ||
| oss_decision.py | ||
| README.md | ||
File Structure
August 27th, 2024
To make it easy to see how calls are transformed for each model/provider:
we are working on moving all supported litellm providers to a folder structure, where folder name is the supported litellm provider name.
Each folder will contain a *_transformation.py file, which has all the request/response transformation logic, making it easy to see how calls are modified.
E.g. cohere/, bedrock/.