* bump: version 1.86.0 → 1.86.1
* chore: refresh uv.lock for 1.86.1
* fix(team): keep team_alias cache in sync on _cache_team_object writes (#28737)
* fix(team): keep team_alias cache in sync on _cache_team_object writes
_cache_team_object wrote only to the team_id:<id> cache key, but the
JWT auth path that uses team_alias_jwt_field reads from a separate
team_alias:<alias> key (get_team_object_by_alias caches under both
keys on miss, but reads only the alias-keyed one). After any
team-mutation endpoint (team_model_add, team_model_delete,
update_team, the two access-group writes) the team_id cache was
refreshed but the team_alias cache stayed stale until TTL — JWT
callers using team_alias_jwt_field kept seeing the pre-mutation
team for the full cache window.
Mirror the write under the alias key inside _cache_team_object so
every existing caller stays in sync without further changes. Skip
the alias write when team_alias is None/empty so we don't collide
across alias-less teams.
Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the
LIT-3244 fix correctly invalidated the team_id cache but the
customer's JWT used team_alias_jwt_field, so they kept hitting the
stale alias-keyed entry.
* fix(team): delete (not overwrite) team_alias cache on _cache_team_object
The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias>
from _cache_team_object. team_alias is NOT unique in the schema
(no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias
enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises).
Writing the alias-keyed cache from the generic refresh path bypassed
that check: a team admin renaming their team to collide with another
team's alias could silently overwrite the cached team for JWT-by-alias
auth, swapping the resolved team under that alias for the cache window.
Switch the alias-keyed operation from a write to a delete (mirroring
the dual-cache delete pattern in _delete_cache_key_object). After every
team write, the next JWT-by-alias reader cache-misses and falls through
to get_team_object_by_alias, which (a) re-fetches the fresh team from
DB, closing the LIT-3244 staleness gap that motivated this PR, and
(b) enforces alias uniqueness before populating either cache key.
team_id:<id> writes are unchanged — team_id is the table PK and is
guaranteed unique.
Surfaced in veria-ai review on #28739.
* fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id
extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)`
which substring-matches the `model_id,` inside the file-ID encoding's
`llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id
then fed that deployment UUID back into the auth path as a model
candidate via _extract_models_from_managed_resource_id, and every
team-BYOK file attach 403'd with:
team not allowed to access model. This team can only access
models=['openai/*']. Tried to access <deployment-uuid>
The team's models list correctly contains the public name (`openai/*`)
that target_model_names matches, but the bogus UUID candidate fails
the wildcard check first.
Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it
matches the legitimate top-level `model_id,<value>` field on
vector_store unified IDs and skips substring matches inside other
fields. File-IDs (which have no top-level `model_id` field) now
return None and contribute no spurious UUID candidate.
Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's
exact flow: team with openai/* BYOK deployment, JWT-scoped user,
POST /v1/vector_stores/{id}/files attaching a file uploaded with
target_model_names=openai/gpt-4o.
* fix(proxy): hydrate wildcard discovery credentials (#28284)
* fix(proxy): hydrate wildcard discovery credentials
* fix(proxy): constrain wildcard credential hydration
* chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)
* chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214
The original account (888602223428) was put under a security restriction by
AWS after a root access key leaked in a PR comment. While that account works
its way through the AWS Support unlock process, Bedrock-touching CI tests have
been migrated to a fresh account (941277531214).
Changes:
- Replace 26 hardcoded references to 888602223428 with 941277531214 across
8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime
ARNs, batch execution role ARN, and example proxy config).
- The provisioned-model and imported-model ARNs are referenced only from
mocked unit tests — no AWS resources to recreate.
- The batch execution IAM role has been recreated in the new account with
the same name and equivalent permissions.
- The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC,
hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account
under the same names — see tools/agentcore-deploy/ in a follow-up.
CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME
were updated separately via the CircleCI API to point at the new account.
Smoke-tested locally against the new account:
aws bedrock-runtime converse --region us-west-2 \
--model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \
--messages '[{"role":"user","content":[{"text":"ping"}]}]'
→ 200, model returned 'pong'
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes
The first migration commit replaced just the account ID, but AgentCore
auto-assigns a random 10-char suffix to every runtime on creation — we
can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the
new account. Updated the AgentCore-runtime ARNs in the three files that
reference real runtime IDs (not the mock-based unit-test ARNs).
Deployed runtimes:
arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp
arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy
Both runtimes are status=READY and pass a smoke invoke:
$ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}'
→ 200, {"result": "echo: ping"}
The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the
deploy artifacts). Tests that only verify the SDK wiring will pass; if any
test asserts on agent output content, swap the echo for the real agent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(tests): point Bedrock batch tests at new-account S3 bucket
The account migration (888602223428 -> 941277531214) was a flat
account-ID swap, which only rewrites ARNs that embed the account
number. S3 bucket names carry no account ID, so the live Bedrock
batch tests still uploaded to `litellm-proxy` — a bucket that lives
in the old account. S3 names are globally unique, and the old account
still holds that name, so it can't be recreated in the new account.
Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees
global uniqueness). The bucket must be created in 941277531214 and the
batch execution role granted s3:GetObject/PutObject/ListBucket on it
before this job is run in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(tests): point live S3 logging test at new-account bucket
Same account-ID-free blind spot as the batch bucket: `load-testing-oct`
lives in the old account and its name can't be reused globally. The
`logging_testing` CI job is wired into the workflow and runs
test_basic_s3_logging, which uploads to this bucket with the CI env
creds, then lists and deletes objects — a live dependency.
Rename to `load-testing-oct-941277531214`. The bucket must exist in the
new account with the CI IAM principal granted
s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(tests): repoint Bedrock guardrail IDs to new-account guardrails
The migration left guardrail IDs untouched (no account ID in them), so
all live guardrail tests failed with "guardrail identifier or version
does not exist" against 941277531214. Recreated both guardrails in the
new account and updated the hardcoded IDs:
- wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD,
with explicit inputAction=ANONYMIZE so masking applies to INPUT,
which is the source litellm's moderation hook sends)
- ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set
to the exact string the tests assert on)
Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the
guardrailConfig in test_bedrock_completion.py. Verified locally: the 5
previously-failing guardrail tests now pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bedrock): migrate legacy models to current inference profiles
The new CI account (941277531214) cannot invoke legacy Bedrock models
(AWS gates them: "marked by provider as Legacy... not actively using in
the last 30 days"). Migrated the live-call tests:
- anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0
- anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0
Current Claude models on Bedrock require the us. inference-profile prefix
(bare on-demand ids are rejected).
cohere.command-r-plus has no working replacement (all Cohere is legacy-
gated in the new account): swapped to claude-haiku-4-5 in provider-
agnostic param lists. amazon.titan-image-generator skipped (no working
replacement). Mocked/transformation/cost tests that reference the legacy
strings are intentionally left unchanged. Verified live against the new
account.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bedrock): repoint SageMaker + Knowledge Base to new-account resources
These referenced account-scoped resources by hardcoded id that only
existed in the old account, so the migration's account-ID swap missed
them. Recreated in 941277531214 and repointed:
- SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614
-> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge)
- Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless
vector store + titan-embed-text-v2, seeded with a LiteLLM doc)
Verified live: test_sagemaker.py (12 passed) and
test_bedrock_knowledgebase_hook.py (12 passed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214)
claude-opus-4-7 is listed in the new Bedrock CI account's foundation
models but invoke is denied (AccessDeniedException: "not available for
this account"). Bedrock access to the flagship Opus requires an AWS
Sales request, not the self-serve model-access toggle, so it can't be
enabled inline with the rest of the account migration.
Add an optional `skip_reason` to ModelEntry and set it on the
bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip.
Cell count (231) and route coverage are unchanged, so the structural
asserts still pass. Restore coverage by deleting the one skip_reason
line once access is granted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bedrock): swap/skip legacy-gated models unavailable on new CI account
The migrated AWS account (941277531214) cannot access several models that
the old account could, so the remaining red CI jobs were hitting real
Bedrock "Access denied / Legacy" and "account not authorized" errors:
- image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is
legacy-gated), matching the existing titan skip.
- batches: skip test_async_file_and_batch (Bedrock batch inference is not
authorized on the new account; requires an AWS support case).
- litellm_overhead: swap legacy claude-3-5-haiku for the active
us.anthropic.claude-haiku-4-5 inference profile.
- test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the
active us.anthropic.claude-sonnet-4-5 inference profile.
https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa
* test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account
- e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference
is not authorized on account 941277531214) and migrate the missed
s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214.
- build_and_test: swap legacy bedrock claude-3-sonnet for the active
us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured
output e2e test.
https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa
* test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (#28791)
Replace the silent skips added for the new CI account with noisier behavior:
- reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present)
instead of skipping, so the missing entitlement stays visible in CI; they
still skip when AWS creds are absent (local dev)
- Bedrock batch inference tests: drop the skip so they run and fail until
batch access is granted
- Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the
transform + cost-tracking path stays under test without live model access
https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT
Co-authored-by: Claude <noreply@anthropic.com>
* test(bedrock): use pytest.xfail for known-failing opus-4-7 cells
Replace pytest.fail with pytest.xfail when a model has a fail_reason,
so known-broken cells stay visible as XFAIL without keeping CI red.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943)
* docs: add one-line docstring to _disable_debugging (#27894)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831)
Squash-merged by litellm-agent from Cyberfilo's PR.
* Sanitize empty text content blocks on /v1/messages (#27832)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint
The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic
Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not
Found. Both AmazonMantleConfig (chat/completions caller route) and
AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded
the wrong path, so every Mantle request 404'd before reaching the model.
Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages
API at /anthropic/v1/messages with SSE streaming."
https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock
Confirmed independently against the live endpoint:
/v1/chat/completions -> 200 OK
/v1/messages -> 404 Not Found (what litellm used)
/anthropic/v1/messages -> 200 OK (Claude only)
Adds a regression test asserting both Mantle configs build the
/anthropic/v1/messages path, and updates the existing assertions that
encoded the wrong path.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
* fix: sanitize empty text blocks in sync anthropic_messages_handler path
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(fireworks_ai): strip thinking_blocks from chat messages before API call
Fireworks OpenAI-compatible ChatMessage schema uses additionalProperties:false
and rejects Anthropic-style messages[].thinking_blocks (e.g. Claude Code replays),
returning invalid_request_error. Remove the field in _transform_messages_helper
alongside provider_specific_fields.
Adds unit test test_transform_messages_helper_strips_thinking_blocks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(fireworks_ai): drop inline comments from message sanitization
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(fireworks_ai): explain why provider_specific_fields and thinking_blocks are stripped
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: strip Gemini thought-signature from tool_use.id in non-streaming path; example websearch config (#27873)
- adapters/transformation.py: mirror the streaming path and strip the
`__thought__<b64>` suffix off `tool_call.id` before building the
AnthropicResponseContentBlockToolUse. Base64's `+ / =` characters
violate Anthropic's `^[a-zA-Z0-9_-]+$` tool_use.id pattern, so when a
conversation that flowed through Gemini is later replayed to an
Anthropic-native provider (Bedrock or Anthropic API) the request 400s.
- example_config_yaml/websearch_interception_config.yaml: register the
interceptor under `callbacks:` not `success_callback:`. `success_callback`
does not run pre-request hooks, so the tool-conversion step never fires
on `/v1/messages` and the raw `web_search_20250305` tool is forwarded
to Bedrock, which 400s.
- adds a unit test pinning the non-streaming strip behavior and the
surviving `^[a-zA-Z0-9_-]+$` shape of the resulting id.
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com>
* Fix/azure image edit auth header (#27863)
* fix(azure/image_edit): use api-key header instead of Authorization Bearer
Delegate `AzureImageEditConfig.validate_environment` to
`BaseAzureLLM._base_validate_azure_environment` so the image-edit route
follows the same auth resolution as every other Azure provider:
- prefer the Azure-native `api-key` header when an API key is available
- fall back to `Authorization: Bearer <azure_ad_token>` only for AAD auth
The previous implementation unconditionally set
`Authorization: Bearer <api_key>`, which is the OpenAI-direct convention
and is rejected by Azure OpenAI / APIM-fronted deployments with
`401 Access denied due to missing subscription key`.
Adds regression tests covering api_key kwarg, litellm_params.api_key, and
the AAD-token fallback path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(azure/image_edit): pin api-key precedence semantics + add regression test
Address review feedback that the move to
``BaseAzureLLM._base_validate_azure_environment`` changed the relative
priority of the positional ``api_key`` kwarg vs. ``litellm_params["api_key"]``.
The new behavior — ``litellm_params["api_key"]`` wins, positional only fills
in when ``litellm_params["api_key"]`` is empty — is intentional and matches
every other Azure ``validate_environment``: ``AzureVideosConfig`` uses the
exact same merge logic, while ``AzureVectorStoresConfig`` and
``AzureResponsesAPIConfig`` don't accept a positional ``api_key`` at all.
The old ``or`` chain (positional wins) was the outlier and was part of the
same OpenAI-vs-Azure convention drift that produced the original
``Authorization: Bearer`` bug.
The only production caller (``llm_http_handler.image_edit``) sources both
values from the same ``litellm_params.api_key``, so this change is
behaviorally a no-op there. Document the precedence in the docstring and
lock it in with an explicit test so future refactors can't quietly
re-invert it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Adam Kirstein <adam.kirstein@disney.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(azure/image_edit): expect api-key header instead of Authorization Bearer
PR #27863 fixed Azure image edit to use the Azure-native api-key header
instead of OpenAI's Authorization: Bearer convention, but did not update
test_azure_image_edit_litellm_sdk to match. The test still asserted
'Authorization' in headers, which now fails since the new code routes
through BaseAzureLLM._base_validate_azure_environment and emits
api-key when an api_key is provided.
Update the assertion to pin the correct Azure behavior: api-key header
present with the resolved key, and no Authorization header.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com>
Co-authored-by: Adam Kirstein <107421694+justalittleadam@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Adam Kirstein <adam.kirstein@disney.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(ollama): Include provider in model list for ollama (#26135)
* Include provider in model names for ollama
* Fix unit tests
* fix(ollama): process both thinking and content in same streaming chunk (#26098)
* fix(health_check): skip max_tokens for image_generation mode (#26417)
* fix(health_check): skip max_tokens for image_generation mode
`_update_litellm_params_for_health_check` injected `max_tokens` for
every deployment. OpenAI `/v1/images/generations` strictly rejects
unknown fields, so health checks for dall-e-* and gpt-image-1 always
failed with `400 "Unknown parameter: 'max_tokens'"` even though the
actual image endpoint calls succeed. Skip the `max_tokens` injection
when `model_info.mode == "image_generation"`. `messages` still gets
injected (downstream `_filter_model_params` already strips it for
non-chat handlers).
* Switch to allow-list with per-deployment override
Per @krrishdholakia review: deny-listing image_generation only re-introduces
the same bug for every other non-chat mode (embedding, audio_*, rerank,
video_generation, ocr, search, moderation, ...).
Replace the single image_generation skip with `_MAX_TOKEN_SUPPORT_MODES =
{chat, completion, responses}`. Missing `mode` is treated as chat for
backward compatibility. New modes are safe by default.
Add `model_info.health_check_supports_max_tokens` as an operator escape
hatch — True forces injection on a non-listed deployment (operator wants
to bound probe tokens), False suppresses it on a chat-style deployment
behind a strict-schema provider.
Tests: parametrize over 3 chat-style + 10 non-chat modes, plus override
on/off and the no-mode legacy path.
* fix(http_handler): handle RequestNotRead in MaskedHTTPStatusError for multipart uploads (#26718)
Squash-merged by litellm-agent from dawidkulpa's PR.
* fix(ollama): guard against double 'ollama/' prefix in live model listing
Greptile flagged that Ollama servers can return names that already start
with 'ollama/'. Check the prefix before prepending so we don't produce
'ollama/ollama/...'. Adds a regression test.
* Fix Ollama empty reasoning stream chunks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: James Myatt <james@jamesmyatt.co.uk>
Co-authored-by: VHash <225398745+vhash0@users.noreply.github.com>
Co-authored-by: hayden <sewhan.kim+@a-bly.com>
Co-authored-by: dawidkulpa <84176950+dawidkulpa@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* chore: reject bare str at file-input sinks to prevent local-file read (#27667)
Squash-merged by litellm-agent from stuxf's PR.
* fix: use os.PathLike in ocr sink and check truthy reasoningSummary for bridge
- ocr/main.py: widen Path check to os.PathLike for consistency with other sinks
- main.py: bridge condition checks truthiness of reasoning_summary, not just None
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: remove unused pathlib.Path import in ocr/main.py
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The enterprise package is installed as `litellm_enterprise` (per
enterprise/pyproject.toml), but several tests imported it as
`enterprise.litellm_enterprise.*` — a path that only resolves
because the repo root happens to sit on sys.path, letting Python's
implicit namespace package machinery discover `enterprise/` as a
directory.
This breaks any test runner that relocates source (e.g. the
mutation-testing workflow, which copies tests under `mutants/`) and
also caused two `patch()` strings to target a module path that does
not match what production code imports — meaning those mocks were
never actually patching the production module's attribute.
Replace `from enterprise.litellm_enterprise.` with the canonical
`from litellm_enterprise.` across 6 test files, and fix two
`patch()` target strings (and one `sys.modules` patch key in the
SSO test) to match.
* feat(bedrock): support retrieve for model-invocation-job batch ARNs
`bedrock.retrieve_batch` previously only handled `:async-invoke/` ARNs
(Twelve Labs Marengo embeddings). The `:model-invocation-job/` ARNs
returned by `CreateModelInvocationJob` (the bulk batch inference API
behind `bedrock.create_batch`) fell through and returned a misleading
data-plane error, leaving created jobs unretrievable through the
LiteLLM batches API.
The two ARN families live on different AWS service endpoints
(`bedrock-runtime` data plane vs `bedrock` control plane), so they need
distinct handlers. This adds:
* `BedrockBatchesHandler._handle_model_invocation_job_status` — calls
the control plane via boto3 (`bedrock:GetModelInvocationJob`),
reusing `BaseAWSLLM.get_credentials` for credential resolution so
model_list / env / role-assumption configs continue to apply. The
response is reshaped into a `LiteLLMBatch` with the same status
mapping `transform_create_batch_response` already uses.
* Output-file-URI prediction. Bedrock surfaces the user-supplied
`s3OutputDataConfig.s3Uri` *prefix* in `GetModelInvocationJob`, but
results actually land at `<prefix>/<job-id>/<basename(input)>.out`.
We compute that single-file URI client-side and surface it as
`output_file_id`, so OpenAI-style `client.files.content(...)` works
without an extra `ListObjectsV2` round-trip. The bare prefix stays
in metadata for callers that want the manifest.
* Dispatch in `litellm/batches/main.py` for the new ARN family,
alongside the existing async-invoke branch.
* Unit tests covering ARN parsing, output-URI prediction (incl. edge
cases), the full status mapping, region resolution precedence, and
failure-message propagation.
Note: `request_counts` is intentionally `(0, 0, 0)` —
`GetModelInvocationJob` does not report per-record counts; getting
accurate numbers requires parsing `manifest.json.out` from the output
S3 prefix, which is left to callers.
Made-with: Cursor
* fix(bedrock): address PR feedback on model-invocation-job retrieve
Addresses Greptile P2 findings on #26834:
1. Use the bare job id (not the full ARN) when constructing the
`api_base` URL for `pre_call` logging. Passing the full ARN double-
counts the `model-invocation-job/` segment and embeds colons in the
path, producing misleading log lines.
2. Drop the `or output_prefix` fallback when `_predict_output_file_uri`
returns None. A bare prefix is not a downloadable object and surfacing
it as `output_file_id` re-creates the very NoSuchKey bug this handler
exists to fix. The bare prefix is still preserved in
`metadata["output_s3_uri"]` for callers that want to do their own S3
listing or read `manifest.json.out`.
`metadata["output_file_uri"]` uses "" rather than None to satisfy the
OpenAI Batch metadata schema (`dict[str, str]`); callers should branch
on the typed `output_file_id` field instead.
Also expands test coverage on the new code path:
- new "stay None" regression test for the prediction-fail case
- pre_call/post_call logging hook assertions (incl. the bare-id URL)
- explicit cancelled_at / expired_at coverage
- _to_epoch type-handling matrix and the boto3 ImportError branch
- defensive _extract_region_from_bedrock_arn exception path
- empty-basename case for _predict_output_file_uri
Patch coverage on the changed lines is now 100% (the only remaining
uncovered lines in the file belong to the pre-existing
`_handle_async_invoke_status` method, which this PR does not touch).
Made-with: Cursor
* test(bedrock): cover retrieve_batch dispatch for both ARN families
Codecov flagged 8 uncovered lines on `litellm/batches/main.py` after
this PR refactored the Bedrock dispatch into a single guard with two
sub-branches (`async-invoke` + `model-invocation-job`). Existing tests
exercised the handlers directly but not the dispatch in `main.py`.
Adds `tests/test_litellm/batches/test_retrieve_batch_bedrock_dispatch.py`
with 6 mocked tests that exercise `litellm.retrieve_batch` end-to-end
for the dispatch logic:
- async-invoke ARN routes to `_handle_async_invoke_status`
- async-invoke ARN with no region falls back to "us-east-1" (preserves
prior behavior on this branch)
- model-invocation-job ARN routes to the new
`_handle_model_invocation_job_status` handler
- model-invocation-job ARN with no region forwards None (so the new
handler can sniff region from the ARN itself, rather than getting
silently routed to us-east-1)
- unrelated bedrock ARN family falls through to the generic
provider-config retrieve path (neither special handler invoked)
- non-bedrock batch ids skip the bedrock dispatch entirely
Both handlers are mocked at the import site so the tests don't hit
AWS — the focus here is purely the new dispatch logic in main.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): move retrieve_batch dispatch test to tests/test_litellm/
The dispatch test landed under `tests/test_litellm/batches/`, a new
directory that no upstream `test-unit-*.yml` workflow's `test-path`
allow-list includes. As a result, the test was never executed in CI
and codecov reported `litellm/batches/main.py` patch coverage at
11.11% (8 lines uncovered) — the lines belonging to this PR's
dispatch refactor itself.
Move the file up one level so it matches the
`tests/test_litellm/test_*.py` glob that `test-unit-misc.yml`
already runs, and adjust `sys.path.insert` for the new depth.
The companion handler tests under
`tests/test_litellm/llms/bedrock/batches/test_handler.py` are
unaffected — they're picked up by the `llms` directory in
`test-unit-llm-providers.yml`.
Made-with: Cursor
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Anthropic rejects tool_use/tool_result when tools is omitted. Always map
and attach the dummy tool in transform_request so CLIs work without
litellm.modify_params.
- Add unit test for transform_request dummy tool with modify_params off
- Adjust parallel function calling integration expectations: Bedrock
Converse still requires modify_params for this path
Co-authored-by: Cursor <cursoragent@cursor.com>
* Refactor Bedrock response stream shape handling
- Introduced a module-level constant `BEDROCK_RESPONSE_STREAM_SHAPE` to cache the response stream shape, eliminating the need for per-instance caching in `BedrockEventStreamDecoderBase`.
- Updated relevant methods to utilize the new constant, improving performance by avoiding redundant loading of the shape.
- Added tests to ensure the shape is loaded correctly at import time and is consistent across different modules.
- Added a new mock server script for testing Bedrock pass-through functionality.
* Refactor response parsing for Bedrock and SageMaker
- Improved code readability by formatting the parsing method calls in `AWSEventStreamDecoder` for both Bedrock and SageMaker response stream shapes.
- Added blank lines for better separation of code blocks in `invoke_handler.py` and `common_utils.py` to enhance maintainability.
* Enhance error handling for Bedrock and SageMaker response stream shape loading
- Wrapped the loading logic in `_load_bedrock_response_stream_shape` and `_load_sagemaker_response_stream_shape` with try-except blocks to gracefully handle exceptions.
- Added logging to warn when the response stream shape cannot be pre-loaded, ensuring the module imports cleanly.
- Updated tests to verify that loading failures return `None` instead of propagating exceptions.
* Implement error handling for missing response stream shapes in Bedrock and SageMaker
- Added checks in `_parse_message_from_event` methods to raise appropriate errors when `BEDROCK_RESPONSE_STREAM_SHAPE` or `SAGEMAKER_RESPONSE_STREAM_SHAPE` is None, ensuring clearer error reporting.
- Updated logging messages to reflect the unavailability of event-stream decoding for both Bedrock and SageMaker.
- Enhanced unit tests to verify that the correct exceptions are raised when the response stream shapes are not loaded.
* fix(hosted_vllm): normalize custom tools for chat completions
Convert custom tool definitions into OpenAI function tools before forwarding hosted_vllm chat requests to avoid provider-side validation failures. Add a regression test and include a local curl verification screenshot.
Made-with: Cursor
* Fix black issue
* Fix hosted vllm custom tool schema fallback
* fix black
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(audio_transcription): add NVIDIA Riva STT provider
Adds nvidia_riva as a new audio transcription provider, supporting both
NVCF-hosted and self-hosted Riva ASR deployments via gRPC streaming.
- Auto-resamples input audio to 16 kHz mono LINEAR_PCM (soundfile + numpy,
audioread fallback) so callers can send any common format.
- Maps OpenAI params: language (en -> en-US), response_format (text/json/
verbose_json), timestamp_granularities=["word"] -> enable_word_time_offsets,
word offsets converted ms -> s for verbose_json.
- Auth: NVCF when nvcf_function_id is set (SSL on by default), self-hosted
otherwise (SSL off by default), with explicit use_ssl override.
- gRPC errors wrapped via NvidiaRivaException -> litellm exception classes.
- Optional deps gated behind [stt-nvidia-riva] extra (nvidia-riva-client,
soundfile, audioread, numpy).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(nvidia_riva): address PR review feedback
- handler: forward call-level `timeout` to streaming_response_generator
(kwarg-detected via inspect for older riva-client compat) so a stalled
Riva server cannot block the caller indefinitely.
- audio_utils: spill bytes to a tempfile before audioread.audio_open;
most audioread backends (FFmpeg, GStreamer) require a real filesystem
path and previously raised TypeError on BytesIO, breaking the mp3/m4a
fallback path.
- audio_utils: prefer soxr / scipy.signal.resample_poly for resampling
(anti-aliased polyphase) when installed, falling back to linear only
as a last resort. Avoids aliasing on 44.1/48 kHz -> 16 kHz downsamples.
- transformation: bare `es` now maps to es-ES (Castilian) instead of
es-US, matching BCP-47 conventions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: trigger CI re-run [stabilize loop 1/3]
* Update litellm/llms/nvidia_riva/audio_transcription/transformation.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* chore: trigger CI re-run [stabilize loop 1/3]
* fix code qa
* fix lint
* fix mypy
* fix mypy
* Fix NVIDIA Riva ASR service lookup
* Fix NVIDIA Riva transcription payload logging
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: oss-pr-review-agent-shin[bot] <281797381+oss-pr-review-agent-shin[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic, mcp): sanitize tool names to match Anthropic's `^[a-zA-Z0-9_-]{1,128}$`
Tool names with characters like `/` or `.` (commonly produced by the
OpenAPI -> MCP generator from `operationId`s such as
`actions/download-job-logs-for-workflow-run`) caused Anthropic to reject
requests with `tools.N.custom.name: String should match pattern
'^[a-zA-Z0-9_-]{1,128}$'`.
Two layers of fix:
1. Anthropic transformation: build a per-request forward map (original ->
sanitized, disambiguated by suffix on collisions) and a reverse map
(only for names actually rewritten). Forward map is applied to tool
defs, `tool_choice`, and historical assistant tool_calls in messages.
Reverse map is threaded through both the non-streaming and streaming
response paths so callers continue to see their original tool names
in `tool_use` blocks.
2. OpenAPI -> MCP generator: sanitize `operationId` (and the
method+path fallback) at registration time so generated MCP tools are
valid for any strict-name provider, not just Anthropic. The dashboard
preview endpoint applies the same sanitization for parity.
Includes unit tests covering: collision disambiguation between
`foo_bar` and `foo/bar` in the same request, reverse-map only firing
for actually-rewritten names, message rewrite for historical tool_calls,
streaming chunk_parser reverse-mapping, and sanitization of OpenAPI
operationIds plus the preview endpoint output.
Made-with: Cursor
* fix(anthropic): build tool-name maps in transform_request, not optional_params
The previous patch stashed the per-request forward and reverse tool-name
maps under ``optional_params["_anthropic_tool_name_forward_map"]`` and
``optional_params["_anthropic_tool_name_map"]``. ``optional_params`` is
the dict that becomes the JSON body via ``data = {**optional_params}``,
so those internal keys leaked over the wire and Anthropic 400'd with:
_anthropic_tool_name_forward_map: Extra inputs are not permitted
Worse, this meant *every* request whose tool list contained any name with
an invalid character (the exact case the patch was meant to fix) regressed
into a confusing meta-error pointing at LiteLLM's internal map instead of
the offending tool.
Fix: move all tool-name sanitization into ``transform_request``, which is
the single chokepoint already shared by ``AnthropicConfig``,
``AmazonAnthropicConfig`` (Bedrock invoke), ``VertexAIAnthropicConfig``,
and ``AzureAnthropicConfig`` (all call ``super().transform_request`` /
``AnthropicConfig.transform_request(self, ...)``). New static helper
``_sanitize_tool_names_in_request`` walks the already-Anthropic-shaped
``optional_params["tools"]`` (only ``type=="custom"`` entries -- hosted
tool names are reserved by Anthropic and must not be touched), builds
the per-request forward/reverse maps, and applies the forward map in
place to ``tools[*].name`` and ``tool_choice.name``. The reverse map is
stashed exclusively on ``litellm_params`` (which is never serialized to
a provider) under ``_anthropic_tool_name_map`` for the response paths
to consume.
Side effect of this restructure: ``map_openai_params`` is now a pure
OpenAI->Anthropic param translator with no side-channel state, which
matches its contract everywhere else in the codebase.
Tests: replaced the now-incorrect "stashes maps in optional_params"
tests with regressions that assert no underscore-prefixed keys appear
in either ``optional_params`` after ``map_openai_params`` or in the
final ``transform_request`` body. Added end-to-end coverage for:
sanitization in ``transform_request``, ``tool_choice`` rewriting,
historical ``tool_calls`` rewriting in messages, and hosted-tool
passthrough.
Made-with: Cursor
* fix(anthropic): always sanitize empty text content blocks
Anthropic 400s on `{"role": "user", "content": ""}` with:
"messages: text content blocks must be non-empty"
LiteLLM already had `_sanitize_empty_text_content` to rewrite empty text
to a placeholder, but it was gated behind `litellm.modify_params=True`.
With that flag off (default), empty content from upstream agent
frameworks (e.g. pydantic-ai) flowed straight through and tripped the
Anthropic validator.
Fix:
- Always run `_sanitize_empty_text_content` at the top of
`anthropic_messages_pt`, independent of `modify_params`. There is no
way to "pass through" an empty text block, so this is non-optional.
The richer tool-call sanitizations (Cases A/B/D, which actually
mutate conversation structure) remain gated on `modify_params`.
- Extend `_sanitize_empty_text_content` to also handle list-of-blocks
content (`[{"type": "text", "text": ""}]`), not just string content.
Adds 3 regression tests covering string content, list-of-blocks
content, and the no-op case (non-empty messages with modify_params off).
Made-with: Cursor
* fix(anthropic): drop dead tool-name forward-map params, fix mypy + caller-mutation
- remove unused `name_forward_map` param from `_map_tool_choice`,
`_map_tool_helper`, `_map_tools` and the `_apply_anthropic_tool_name_forward`
helper. Production sanitization runs in `_sanitize_tool_names_in_request`
at `transform_request`; these params were never threaded through.
- handler.py: use `ANTHROPIC_TOOL_NAME_REVERSE_MAP_KEY` constant instead of
the hardcoded `"_anthropic_tool_name_map"` string.
- fix mypy `"object" has no attribute "__iter__"` in
`_rewrite_tool_names_in_messages` by guarding `tool_calls` with
`isinstance(..., list)`.
- `_sanitize_tool_names_in_request`: build a new tools list with copy-on-
change entries (and copy `tool_choice` on rewrite) so a caller reusing
the same tool list/dicts across requests doesn't see its inputs
permanently rewritten.
- doc-comment `_build_request_tool_name_maps` clarifying it operates on
OpenAI-format tools (vs `_sanitize_tool_names_in_request` which runs
on Anthropic-format tools post-`_map_tools`).
- tests: drop 3 tests pinning the now-removed param paths; add coverage
for tool_calls + None function_call rewrite and caller-dict immutability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(mcp): inherit stored credentials in test/tools/list for edit flow
When editing an existing MCP server, the Tool Configuration preview
calls POST /mcp-rest/test/tools/list with server_id but no credentials
(management API redacts them). The endpoint now calls
_inherit_credentials_from_existing_server() so stored bearer tokens
and OAuth2 M2M credentials are loaded from global_mcp_server_manager
automatically — tools load without re-entering credentials.
New servers (no server_id) and requests with explicit credentials are
unaffected (function is a no-op in both cases).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(mcp): show all tools in edit panel, not just allowed tools
Edit flow was passing externalTools (from GET /tools/list, filtered by
allowed_tools) to MCPToolConfiguration, disabling the internal hook.
Remove the external props so the internal hook fires via
POST /test/tools/list, which returns all tools unfiltered. Combined
with the credential inheritance fix, tools load automatically without
re-entering credentials and all tools are visible for re-configuration.
existingAllowedTools still pre-checks previously allowed tools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix order-dependent collision in _build_anthropic_tool_name_maps
Use a two-pass approach: first pre-register all already-valid tool names
in the 'used' set, then sanitize/disambiguate names that need rewriting.
This ensures valid names always have priority regardless of input order,
preventing duplicate tool names on the wire when e.g. 'foo/bar' appears
before 'foo_bar' in the tool list.
Add regression test for the reversed ordering case.
* Fix OpenAPI tool name collision: disambiguate sanitized names with numeric suffixes
sanitize_openapi_tool_name replaces all invalid chars with '_', but when
two operationIds differ only by sanitized characters (e.g. 'foo/list' and
'foo.list' both become 'foo_list'), the second registration silently
overwrites the first in the tool registry.
Add collision disambiguation in register_tools_from_openapi that appends
_2, _3, ... suffixes when a sanitized name is already taken, mirroring
the existing logic in _build_anthropic_tool_name_maps.
* Fix preview endpoint missing collision disambiguation for tool names
Add used_names tracking and _2/_3 suffix disambiguation to
_preview_openapi_tools, matching the logic in register_tools_from_openapi.
Without this, two operationIds that sanitize to the same string (e.g.
'foo/list' and 'foo.list' both becoming 'foo_list') would show duplicate
names in the preview while registration would disambiguate them.
* Align preview HTTP method order with register_tools_from_openapi
The preview endpoint and register_tools_from_openapi both use
order-dependent collision disambiguation (_2, _3 suffixes). When the
iteration order differs, two operations on the same path with sanitized
names that collide get different suffixes in preview vs registration,
so the dashboard shows names that don't match what actually got
registered.
Also adds a regression test that fails on the swapped order.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Skip duplicate originals in _build_anthropic_tool_name_maps
If the same invalid tool name appeared twice in original_names (e.g.
['foo/bar', 'foo/bar']), the second occurrence overwrote the forward
map entry with a freshly-suffixed name (foo_bar_2), leaving foo_bar
orphaned in 'used' with no reverse mapping. _sanitize_tool_names_in_request
then rewrote both tool entries to foo_bar_2, and Anthropic 400'd on
duplicate tool names.
Skip the rewrite if forward already has the original mapped.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(realtime): OpenAI Realtime GA support and beta compatibility
- Normalize beta-style session.update to GA for upstream OpenAI; optional GA→beta
event translation when client sends OpenAI-Beta: realtime=v1
- Default upstream WebSocket without OpenAI-Beta; forward header when client opts in
- Extend OpenAI realtime types for GA event names and conversation item shapes
- Relax LiteLLMRealtimeStreamLoggingObject.results to List[Any] for GA events
- Update proxy client_secrets fallback to omit beta header; dashboard RealtimePlayground
- Add unit tests for remap, translation, and beta header helper
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix results
* fix greptile
* Fix mypy issues
* Remove unused class constants _GA_TEXT_DELTA_TYPES and _GA_AUDIO_DELTA_TYPES
These frozensets were defined as class-level constants in realtime_streaming.py
but never referenced anywhere in the codebase. Removing dead code.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(realtime): use GA-shaped session.update in guardrail injections
The guardrail VAD injection code sent a beta-style session.update with a
flat turn_detection field:
{"session": {"turn_detection": {"create_response": false}}}
When the upstream OpenAI backend operates in GA mode (no OpenAI-Beta
header forwarded), it requires the nested GA shape:
{"session": {"type": "realtime", "audio": {"input": {"turn_detection": {"create_response": false}}}}}
The _remap_beta_session_to_ga helper was only applied to client-
originated session.update messages in client_ack_messages. Internally-
generated session.updates (sent via _send_to_backend) in two paths:
- _handle_raw_backend_message (raw/no provider_config path, line 518)
- backend_to_client_send_messages provider_config path (line 481)
bypassed the remap, so GA upstreams ignored or rejected them, breaking
audio transcription guardrails for all non-beta clients.
Fix: add _make_disable_auto_response_message() helper that always emits
the correct GA-shaped session.update, and replace both injection sites
with it.
Update existing tests to assert the GA nested shape instead of the old
flat beta shape, and add a new unit test for the helper itself.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* Log realtime session type
* Fix beta realtime session payloads
* Fix realtime audio format remapping edge case
* Fix Azure realtime beta session shape
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(BaseAWSLLM): implement shared IAM cache and static credential caching
- Introduced a process-wide shared IAM cache to optimize credential management across instances.
- Added a method to handle caching of static credentials, ensuring only long-lived credentials are cached.
- Updated the get_credentials method to utilize the new caching mechanism for static credential flows.
- Enhanced unit tests to verify the correct behavior of the shared cache and static credential usage.
* refactor(BaseAWSLLM): enhance IAM credential caching and update related tests
- Improved the process-wide IAM credential caching mechanism to better handle static and AssumeRole credentials.
- Renamed the caching method for clarity and updated comments to reflect the new caching behavior.
- Added a fixture to ensure the IAM cache is flushed between tests to prevent leakage of cached entries.
- Updated unit tests to verify the correct behavior of the shared IAM cache, particularly for static credentials and role assumptions.
* refactor(BaseAWSLLM): clarify IAM credential caching behavior and enhance tests
- Updated documentation to specify that only static and ambient environment credentials are cached, excluding AssumeRole and other credential types.
- Modified the caching logic to ensure that AssumeRole credentials are not stored in the IAM cache, requiring STS calls for each request.
- Enhanced unit tests to verify that AssumeRole credentials are not cached and to ensure proper behavior of the IAM cache across different scenarios.
* Code Readability improvement for aws auth path
* refactor(BaseAWSLLM): enhance IAM credential caching documentation and add tests
- Updated comments to clarify the behavior of the in-process IAM credential cache, specifying the TTL for static and ambient credentials.
- Added new unit tests to verify the caching behavior for ambient environment credentials across instances and ensure that static access key sessions are constructed only once when cached.
- Ensured that temporary session tokens and AWS profiles are not cached, validating the expected behavior through additional tests.
* refactor(BaseAWSLLM): improve IAM credential handling and add tests for role assumption
- Updated comments to clarify the behavior of IAM credential caching, particularly regarding the handling of ambient credentials and role assumptions.
- Enhanced unit tests to verify that the caching mechanism correctly distinguishes between already running roles and new role assumptions, ensuring that cached environment credentials are not reused incorrectly.
- Added a new test case to validate the behavior when switching roles, confirming that the system correctly uses AssumeRole when the role changes.
- Add image_generation/http_utils.azure_deployment_image_generation_json_body; call
from azure.py (keeps AzureChatCompletion focused on chat).
- Rename finalize_image_edit_multipart_data to finalize_image_edit_request_data with
docstring covering multipart and JSON POST payloads (review feedback).
Co-authored-by: Cursor <cursoragent@cursor.com>
Substantial reduction (~765 LOC) without changing the security
boundary:
* Drop ContainerOwnershipStore and LiteLLMSkillsStore — both were
one-method-per-Prisma-call wrappers. Inline the calls instead,
matching the established pattern in vector_store_endpoints,
agent_endpoints, and mcp_server/db.py.
* Drop the prisma_client is None in-memory fallback. Production
deploys always have Prisma; running ownership-critical paths on a
process-local dict is a security footgun in the dev-mode case it
was meant to support, and complicates every code path with a
branch. Fail-secure: skip recording if Prisma is unavailable, and
treat reads as "not found" (admin-only).
* Drop the hand-rolled module-level cache. Replace with the existing
litellm.caching.in_memory_cache.InMemoryCache, which already has
TTL + max-size + eviction tested in its own module. Sentinel string
for negative caching since InMemoryCache can't disambiguate "miss"
from "cached as None".
* Tests: drop coverage for removed code paths (in-memory fallback,
hand-rolled cache internals). Keep tests for actual behavior (cache
hit-rate, negative caching, owner check, list filtering,
identity-less reject, admin bypass).
Two cleanups:
* ``LiteLLMSkillsHandler.create_skill`` raised ``HTTPException`` for
identity-less callers, importing FastAPI from a ``litellm/llms/``
module — that violates the project rule that FastAPI lives only
under ``proxy/``. Switch to ``ValueError`` (the same shape the rest
of the handler uses for not-found/forbidden) and update the test.
* The proxy-auth body bouncer derived its observability ban list from
``_supported_callback_params`` only, missing
``_request_blocked_callback_params`` (where ``gcs_bucket_name`` and
``gcs_path_service_account`` live). Two recently-merged sibling PRs
(#27019 added the deny list, #27081 added the test asserting these
are rejected at the request body root) crossed without folding them
together. Union the GCS deny list into the bouncer's derivation so
the single source of truth covers both code paths.
UNSCOPED_RESOURCE_OWNER_SCOPE collapsed every caller without an
identity field (no user_id / team_id / org_id / api_key / token) into
a single shared owner — a cross-tenant access primitive: any two such
callers could see and delete each other's containers and skills.
Drop the sentinel. ``get_primary_resource_owner_scope`` returns
``None`` and ``get_resource_owner_scopes`` returns ``[]`` for
identity-less callers. ``record_container_owner`` and
``LiteLLMSkillsHandler.create_skill`` now reject creates from
identity-less callers with a 403 instead of stamping the placeholder.
Read paths already deny ``owner is None`` correctly so legacy rows
(if any) are admin-only.
LITELLM_ALLOW_UNTRACKED_CONTAINER_ACCESS and
LITELLM_ALLOW_UNOWNED_SKILL_ACCESS were operator-toggleable opt-outs
for the cross-tenant access primitive this PR closes — flipping either
on re-enabled exactly the VERIA-20 read path. Default-secure with no
escape hatch matches sibling fixes (vector-store cred isolation, semantic
cache key isolation, user_config strip): all rejected the
opt-out-of-security pattern.
Untracked containers and unowned skills (rows that pre-date this
enforcement) are admin-only. Non-admin owners need to either re-create
via the now-tracked flow or have an admin assign ``created_by`` on the
existing row. Update tests to assert the strict-only behaviour.
Two cleanups from the /simplify pass:
* ``_CONTAINER_OWNER_CACHE`` and ``_SKILL_CACHE`` now LRU-evict via
``OrderedDict.popitem(last=False)`` instead of full ``clear()`` at
capacity. Full clears converted a steady-state cached workload into a
periodic full-DB-load oscillation as the cache repopulated from zero
and cleared again. Reads now ``move_to_end`` so the just-touched
entry survives the next eviction. Mirrors the pre-existing LRU
pattern in ``_remember_container_owner``.
* ``LiteLLM_ManagedObjectTable.file_purpose`` Literal now includes
``"container"`` so Pydantic validation accepts rows written by the
ownership store.
Strip out the explanatory and historical comments that don't carry
business-logic justification. Comments that simply narrate what code
does — or that explain prior behavior, what was changed, or which PR
introduced a fix — are removed. Docstrings are reduced to a one-line
summary where the long form repeated information already evident from
the code or test data.
No code-behavior changes. All 643 affected unit tests still pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
xAI's chat completions API accounts reasoning_tokens separately from
completion_tokens, but rolls them into total_tokens. This breaks the
OpenAI invariant total_tokens == prompt_tokens + completion_tokens
that downstream consumers (including litellm's own _usage_format_tests
in tests/llm_translation/base_llm_unit_tests.py:58) rely on.
Live capture (grok-3-mini-beta, 2026-05-04):
prompt=14, completion=10, total=336, reasoning=312
14 + 10 = 24, NOT 336.
OpenAI's o1/o3 reasoning models include reasoning_tokens in
completion_tokens, leaving the prompt+completion=total invariant
intact. xAI deviates. This patch aligns xAI to OpenAI semantics by
folding reasoning_tokens into completion_tokens after the parent
OpenAI parser runs.
The fold is idempotent and defensive:
- Only fires when total_tokens == prompt_tokens + completion_tokens
+ reasoning_tokens (the documented xAI shape). Refuses to fold if
the gap doesn't match, guarding against silent corruption when xAI
changes accounting.
- Skips if completion_tokens already covers the gap (already
normalised — e.g. cost calc replays a previously-folded Usage).
xai.cost_calculator.cost_per_token already added reasoning_tokens to
the visible completion count for billing. Post-fold the Usage block
now satisfies that invariant directly, so the cost calc would
double-bill. Updated cost_per_token to detect the OpenAI-normalised
shape (total == prompt + completion) and skip the reasoning add-on
in that case, falling through to the legacy raw-shape behaviour for
callers that bypass the transformation (e.g. proxy log replay).
Tests:
- Adds TestXAIReasoningTokenFolding covering: gap-explained-fold,
idempotent-no-double-fold, no-reasoning-skip, gap-mismatch-skip.
- Adds test_already_normalised_usage_does_not_double_count_reasoning
to lock the cost-calc idempotency.
- Updates 7 pre-existing cost-calc tests whose total_tokens was
internally inconsistent (used the OpenAI-normalised total but kept
reasoning_tokens external) to use the documented xAI raw shape
total = prompt + visible completion + reasoning. Pre-existing
values masked the missing-fold by accident.
Verified end-to-end against the live xAI API:
LITELLM_LOCAL_MODEL_COST_MAP=False (CI default) +
XAI_API_KEY set +
pytest tests/llm_translation/test_xai.py::TestXAIChat::test_prompt_caching
-> PASSED in 18.81s (was: AssertionError on
usage.total_tokens == usage.prompt_tokens + usage.completion_tokens)
20/20 tests in tests/test_litellm/llms/xai/test_xai_cost_calculator.py
and 8/8 in tests/test_litellm/llms/xai/test_xai_chat_transformation.py
pass.
CI runs without LITELLM_LOCAL_MODEL_COST_MAP=True, so litellm.model_cost
is loaded from main-branch JSON (default model_cost_map_url) instead of
the PR's checked-out model_prices_and_context_window.json. Tests that
assert per-model flags added in this PR (supports_max_reasoning_effort,
supports_xhigh_reasoning_effort) therefore pass locally but fail in CI
with 'AssertionError: assert False is True' on 5 cases:
- test_anthropic_model_supports_effort_param_recognizes_supporting_models
[anthropic.claude-mythos-preview, bedrock/.../mythos-preview,
claude-opus-4-5-20251101]
- test_supports_effort_level_handles_provider_prefixes
[bedrock/invoke/us.anthropic.claude-sonnet-4-6-max-True,
claude-sonnet-4-6-max-True]
Add an autouse fixture at tests/test_litellm/llms/anthropic/chat/conftest.py
that monkey-patches litellm.model_cost to the PR-local JSON for every test
in this directory. The parent conftest already snapshots+restores
litellm.model_cost per-function, so the mutation is contained.
This is a scoped workaround. The proper fix is to set the env var
globally in the test workflow once the ~10 inline self-set test files
are audited; tracking that as a follow-up issue.
The chat completion path (`_apply_output_config`) and the /v1/messages
pass-through (`AnthropicMessagesConfig._translate_reasoning_effort_to_anthropic`)
both gate `max` / `xhigh` per model. The two sites had diverged from
near-identical copies into separately maintained blocks, creating a real
drift risk when a new model tier (e.g. Claude 4.8) lands -- a contributor
could update one site and miss the other.
Centralise the gating in `AnthropicConfig._validate_effort_for_model`,
which returns an error message string or `None`. Each call site keeps
its own provider-appropriate exception type (`BadRequestError` for the
chat path, `AnthropicError` for the /v1/messages pass-through) but the
gating decision now comes from one place. Net -11 LOC.
Adds a parametrised unit test exercising the helper directly across
4.5 / 4.6 / 4.7 model families and `max` / `xhigh` / lower-effort
inputs. Existing tests at both call sites continue to pass unchanged.
Addresses Greptile finding on PR #27074.
`azure_ai` is registered in `litellm.openai_compatible_providers`, so
`add_provider_specific_params_to_optional_params` (litellm/utils.py)
auto-stuffs any non-OpenAI kwarg (e.g. `output_config={"effort": "..."}`)
into `optional_params["extra_body"]`. `AzureAnthropicConfig.transform_request`
then strips `extra_body` entirely on the way out, silently dropping the
param — and `AnthropicConfig._apply_output_config` never sees it, so
`effort="invalid"` / `effort="xhigh"` on a non-supporting model
quietly reaches the model with default behavior instead of returning a
clean 400 (as the native `anthropic` provider does).
Promote the keys back to top-level `optional_params` (using `setdefault`
so explicit top-level values win) before delegating to the parent
`AnthropicConfig`. Apply in both `validate_environment` and
`transform_request` so flag detection (`is_mcp_server_used`, etc.) and
output-config validation both run.
Surfaced by the QA matrix expansion on PR #27074: 20 cells where Azure
returned 200 while `anthropic` returned 400 — all `output_config` mode
across haiku_4_5, sonnet_4_5, opus_4_5, sonnet_4_6, opus_4_6, opus_4_7
families with `effort` in {invalid, xhigh, max, low, medium, high}.
Tests:
* `test_output_config_promoted_from_extra_body`: valid effort reaches data
* `test_invalid_output_config_effort_raises_via_extra_body`: 400 on bad effort
* `test_unsupported_effort_xhigh_raises_via_extra_body`: 400 on xhigh-on-Sonnet-4.6
* `test_extra_body_promotion_does_not_clobber_top_level`: setdefault semantics
- Image generation tests patch HTTPHandler.post / get_async_httpx_client so
make_*_azure_httpx_request runs and wire json is asserted on call kwargs.
- Azure image edit: strip model in finalize_image_edit_multipart_data using the
same URL string the handler passes to POST (no second get_complete_url in
transform). BaseImageEditConfig default finalize is a no-op.
Co-authored-by: Cursor <cursoragent@cursor.com>
Azure OpenAI routes image gen/edit by deployment in the URL; sending the
deployment id in model breaks gpt-image-2 (invalid_value). Strip model from
JSON for deployments/.../images/generations and from multipart data for
.../images/edits. Non-deployment URLs (e.g. Azure AI FLUX) unchanged.
Fixes#26316.
Co-authored-by: Cursor <cursoragent@cursor.com>
- claude-sonnet-4-6 + reasoning_effort=max no longer 400s. Renamed
_is_opus_4_6_model to _is_claude_4_6_model at three sites and added
supports_max_reasoning_effort: true to 12 model entries in the JSON
cost map (10 sonnet 4.6 ids + OpenRouter opus 4.6/4.7).
- _map_reasoning_effort now raises BadRequestError(400) directly with
llm_provider, instead of letting Databricks (and similar callers)
surface its raw ValueError as a 500.
- output_config.effort on Opus 4.5 over Bedrock no longer 400s for
missing effort-2025-11-24 beta. Flipped JSON to "effort-2025-11-24"
for bedrock + bedrock_converse and added an auto-attach branch in
_process_tools_and_beta for non-adaptive Anthropic + output_config
on Converse.
- reasoning_effort=xhigh / =max on legacy budget-mode models
(Haiku 4.5, Sonnet 4.5, Opus 4.5) now map to thinking.budget_tokens
8192 / 16384 instead of returning 400. Added two constants in
litellm/constants.py.
Tests updated for all four flips. Validated end-to-end via 306-cell
live proxy matrix (6 model families x 3 routes x 17 effort cases),
all pass.
