* feat(bedrock): support retrieve for model-invocation-job batch ARNs
`bedrock.retrieve_batch` previously only handled `:async-invoke/` ARNs
(Twelve Labs Marengo embeddings). The `:model-invocation-job/` ARNs
returned by `CreateModelInvocationJob` (the bulk batch inference API
behind `bedrock.create_batch`) fell through and returned a misleading
data-plane error, leaving created jobs unretrievable through the
LiteLLM batches API.
The two ARN families live on different AWS service endpoints
(`bedrock-runtime` data plane vs `bedrock` control plane), so they need
distinct handlers. This adds:
* `BedrockBatchesHandler._handle_model_invocation_job_status` — calls
the control plane via boto3 (`bedrock:GetModelInvocationJob`),
reusing `BaseAWSLLM.get_credentials` for credential resolution so
model_list / env / role-assumption configs continue to apply. The
response is reshaped into a `LiteLLMBatch` with the same status
mapping `transform_create_batch_response` already uses.
* Output-file-URI prediction. Bedrock surfaces the user-supplied
`s3OutputDataConfig.s3Uri` *prefix* in `GetModelInvocationJob`, but
results actually land at `<prefix>/<job-id>/<basename(input)>.out`.
We compute that single-file URI client-side and surface it as
`output_file_id`, so OpenAI-style `client.files.content(...)` works
without an extra `ListObjectsV2` round-trip. The bare prefix stays
in metadata for callers that want the manifest.
* Dispatch in `litellm/batches/main.py` for the new ARN family,
alongside the existing async-invoke branch.
* Unit tests covering ARN parsing, output-URI prediction (incl. edge
cases), the full status mapping, region resolution precedence, and
failure-message propagation.
Note: `request_counts` is intentionally `(0, 0, 0)` —
`GetModelInvocationJob` does not report per-record counts; getting
accurate numbers requires parsing `manifest.json.out` from the output
S3 prefix, which is left to callers.
Made-with: Cursor
* fix(bedrock): address PR feedback on model-invocation-job retrieve
Addresses Greptile P2 findings on #26834:
1. Use the bare job id (not the full ARN) when constructing the
`api_base` URL for `pre_call` logging. Passing the full ARN double-
counts the `model-invocation-job/` segment and embeds colons in the
path, producing misleading log lines.
2. Drop the `or output_prefix` fallback when `_predict_output_file_uri`
returns None. A bare prefix is not a downloadable object and surfacing
it as `output_file_id` re-creates the very NoSuchKey bug this handler
exists to fix. The bare prefix is still preserved in
`metadata["output_s3_uri"]` for callers that want to do their own S3
listing or read `manifest.json.out`.
`metadata["output_file_uri"]` uses "" rather than None to satisfy the
OpenAI Batch metadata schema (`dict[str, str]`); callers should branch
on the typed `output_file_id` field instead.
Also expands test coverage on the new code path:
- new "stay None" regression test for the prediction-fail case
- pre_call/post_call logging hook assertions (incl. the bare-id URL)
- explicit cancelled_at / expired_at coverage
- _to_epoch type-handling matrix and the boto3 ImportError branch
- defensive _extract_region_from_bedrock_arn exception path
- empty-basename case for _predict_output_file_uri
Patch coverage on the changed lines is now 100% (the only remaining
uncovered lines in the file belong to the pre-existing
`_handle_async_invoke_status` method, which this PR does not touch).
Made-with: Cursor
* test(bedrock): cover retrieve_batch dispatch for both ARN families
Codecov flagged 8 uncovered lines on `litellm/batches/main.py` after
this PR refactored the Bedrock dispatch into a single guard with two
sub-branches (`async-invoke` + `model-invocation-job`). Existing tests
exercised the handlers directly but not the dispatch in `main.py`.
Adds `tests/test_litellm/batches/test_retrieve_batch_bedrock_dispatch.py`
with 6 mocked tests that exercise `litellm.retrieve_batch` end-to-end
for the dispatch logic:
- async-invoke ARN routes to `_handle_async_invoke_status`
- async-invoke ARN with no region falls back to "us-east-1" (preserves
prior behavior on this branch)
- model-invocation-job ARN routes to the new
`_handle_model_invocation_job_status` handler
- model-invocation-job ARN with no region forwards None (so the new
handler can sniff region from the ARN itself, rather than getting
silently routed to us-east-1)
- unrelated bedrock ARN family falls through to the generic
provider-config retrieve path (neither special handler invoked)
- non-bedrock batch ids skip the bedrock dispatch entirely
Both handlers are mocked at the import site so the tests don't hit
AWS — the focus here is purely the new dispatch logic in main.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): move retrieve_batch dispatch test to tests/test_litellm/
The dispatch test landed under `tests/test_litellm/batches/`, a new
directory that no upstream `test-unit-*.yml` workflow's `test-path`
allow-list includes. As a result, the test was never executed in CI
and codecov reported `litellm/batches/main.py` patch coverage at
11.11% (8 lines uncovered) — the lines belonging to this PR's
dispatch refactor itself.
Move the file up one level so it matches the
`tests/test_litellm/test_*.py` glob that `test-unit-misc.yml`
already runs, and adjust `sys.path.insert` for the new depth.
The companion handler tests under
`tests/test_litellm/llms/bedrock/batches/test_handler.py` are
unaffected — they're picked up by the `llms` directory in
`test-unit-llm-providers.yml`.
Made-with: Cursor
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): map chat tool_choice to Responses API when bridging from completions
OpenAI /v1/responses rejects tool_choice.function. Normalize forced-function
choice from chat shape to {type, name} in LiteLLMResponsesTransformationHandler.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): strip tool_choice.function when top-level name is set
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Anthropic rejects tool_use/tool_result when tools is omitted. Always map
and attach the dummy tool in transform_request so CLIs work without
litellm.modify_params.
- Add unit test for transform_request dummy tool with modify_params off
- Adjust parallel function calling integration expectations: Bedrock
Converse still requires modify_params for this path
Co-authored-by: Cursor <cursoragent@cursor.com>
Reject fnmatch wildcards on non-scope claims when the claim string contains
whitespace so malformed iss values cannot match patterns like trusted.*.
Merge every entry when team_id_jwt_field resolves to a list instead of
keeping only the first element.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Extend responses_api_bridge_check when reasoning_effort + summary aliases
(including nested extra_body) without tools
- Merge summary into reasoning_effort for responses bridge; helpers in utils
- Strip summary aliases in GPT-5 chat mapping when not bridged
- Tests for bridge + merge behavior
Co-authored-by: Cursor <cursoragent@cursor.com>
Stripping body-supplied tags in apply_client_tag_policy_pre_auth silently
disabled per-tag budget enforcement for non-opted-in keys — pre-PR
behavior was that those tags reached _tag_max_budget_check inside
common_checks. The post-auth strip in add_litellm_data_to_request
continues to remove unauthorized tags before they leave the proxy.
Also moves the LiteLLMProxyRequestSetup import to module-level (no
circular dep with auth/user_api_key_auth.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The x-litellm-tags header was merged into request metadata only after the
auth chain completed, so _tag_max_budget_check (which reads tags from the
request body) silently failed open for header-tagged requests — spend
accumulated past max_budget without any 400 budget_exceeded response.
Move the client-tag policy (strip-or-merge gated on allow_client_tags) to
run before common_checks so header tags are visible to budget enforcement.
The post-auth strip+merge in add_litellm_data_to_request stays as
defense-in-depth; the new pre-auth helper is idempotent with it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mcp): forward extra_headers for OpenAPI MCP tools
OpenAPI-generated tools only applied static closure headers and BYOK
Authorization via ContextVar. Copy MCPServer.extra_headers from the
incoming MCP request into _request_extra_headers (set in server.py before
local tool dispatch), merge in openapi_to_mcp_generator via a small helper.
OAuth2 M2M: do not forward caller Authorization from raw_headers (same rule
as _prepare_mcp_server_headers for managed MCP).
Adds TestRequestExtraHeaders and clarifies mcp_server_manager registration
comment.
Fixes#26794
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): access has_client_credentials on MCPServer directly
Greptile: getattr default was redundant; property exists on MCPServer and
mcp_server is non-None inside the extra_headers forwarding block.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The previous detection treated any model with input_cost_per_image
or output_cost_per_image as image generation. Several chat and
embedding models carry those fields to price multimodal vision input,
not generated images:
- gemini-3.1-pro-preview (mode=chat) has output_cost_per_image=0.00012
alongside input/output token pricing.
- azure/gpt-realtime-* (mode=chat) has input_cost_per_image=5e-6.
- amazon.titan-embed-image-v1 (mode=embedding) has
input_cost_per_image=6e-5.
For these models the image-gen branch fired first and reserved a
fraction of a cent per request, short-circuiting the token-priced
path entirely. Long Gemini chats reserved 1 × $0.00012 instead of
the true token cost.
Gate strictly on mode in {"image_generation", "image_edit"}. All 197
real image_generation entries and all 31 image_edit entries
(Flux Kontext, Stability inpaint/outpaint, etc.) carry the right mode,
so the field-presence fallback was unnecessary.
Adds regression tests for the chat-model-with-image-cost-field case
and for image_edit reservation.
Image-generation routes (dall-e-3, flux, etc.) have no per-token output
cost so they fell through to the no-reservation read-time-only path.
Concurrent image requests against a depleted budget could all pass
common_checks (counter exactly at max_budget passes the strict-`>`
gate) and reach the provider before reconciliation caught up.
Add per-image reservation in _estimate_request_max_cost_for_model:
when the model has a per-image cost field, reserve `n × cost_per_image`
upfront. The atomic counter increment serializes concurrent admissions,
so the second request sees the post-first-reservation counter and
raises BudgetExceededError instead of silently leaking through.
Both `output_cost_per_image` and `input_cost_per_image` are honored —
naming is inconsistent across providers (OpenAI dall-e-3 uses
input_cost_per_image, aiml/dall-e-3 uses output_cost_per_image for
the same per-generated-image price).
Per-pixel pricing (DALL-E 2 size variants) and TTS/STT routes still
fall through to read-time enforcement; those are follow-ups.
- Introduce RoutingPrismaWrapper that transparently routes read operations (find_*, count, group_by, query_raw, query_first) to a reader endpoint while writes remain on the writer, enabling Aurora-style reader/writer endpoint splits
- Add IAMEndpoint dataclass and parse_iam_endpoint_from_url() to capture static connection fields from a reader URL so only the IAM token needs to rotate, avoiding the need for separate DATABASE_HOST_READ_REPLICA/etc. env vars
- Enhance PrismaWrapper with per-instance knobs (db_url_env_var, iam_endpoint, recreate_uses_datasource, log_prefix) so writer and reader wrappers are independent: the reader writes its fresh URL to DATABASE_URL_READ_REPLICA and passes datasource override to Prisma since Prisma only auto-reads DATABASE_URL
- Fix deadlock in PrismaWrapper.__getattr__: when called from inside a running event loop, schedule the token refresh as a background task instead of blocking with run_coroutine_threadsafe + future.result(), which would deadlock the loop thread waiting for a coroutine that needs the loop to run
- Fix botocore crash when DATABASE_PORT is unset by defaulting to "5432" in both proxy_cli.py and PrismaWrapper.get_rds_iam_token(); passing None caused botocore to embed the literal string "None" in the presigned URL
- Implement graceful reader degradation: reader connect/recreate failures are non-fatal; wrapper sets _reader_unavailable=True and silently routes reads to the writer to keep the proxy serving traffic during transient reader outages
- Add PrismaClient.writer_db property so the reconnect smoke-test always validates the writer engine specifically; query_raw on the routing wrapper would route to the reader and not verify the newly-recreated writer
- Expose DATABASE_URL_READ_REPLICA in Helm chart (values.yaml + deployment.yaml) via both plain value and secret key reference, and document the field in docker-compose.yml
- Add 887-line test suite covering routing logic, IAM token refresh paths, reader degradation scenarios, datasource override behavior, and the deadlock regression
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
reserve_budget_for_request fell back to reserving the entire remaining
team/key/user headroom whenever a request omitted max_tokens, which
pinned the spend counter at max_budget for the duration of the
in-flight request and false-positive-blocked every concurrent or
back-to-back request until the success callback reconciled. Surfaced
as an integration-test team being budget-blocked at its $2000 cap
while DB spend was $0.144.
Switch the missing-max_tokens path to a fixed default of 16384 output
tokens (mirrors parallel_request_limiter_v3's DEFAULT_MAX_TOKENS_ESTIMATE
precedent), and clamp explicit max_tokens at the model's
max_output_tokens for reservation accounting only. The outbound request
body is unchanged, so providers see whatever the caller actually sent;
only the local integer used to compute reservation cost is bounded.
This also prevents a hostile max_tokens=999999999 from inflating one
request's reservation up to the entire team headroom.
For Opus 4.7 (output $25/M, max_output 128K) on a $2000 budget the
worst-case per-request reservation drops from "everything left" to
$3.20, raising admittable concurrency from 1 to ~625.
Operators upgrading past 35bbca60b0 (which made /metrics auth
default-on) see "Malformed API Key passed in. Ensure Key has 'Bearer '
prefix." with no hint that
litellm_settings.require_auth_for_metrics_endpoint: false restores the
previous unauthenticated behavior. Append that discovery hint to the
existing 401 body so a Prometheus scraper that breaks after upgrade
has a clear migration path. No behavior change.
* feat(sso): show full IdP claims in /sso/debug/callback
The debug callback only displayed the proxy-parsed OpenID summary, so
customers couldn't verify what custom claims (team_id, team_alias, roles,
etc.) the IdP was actually returning. Render two new sections — Raw
Claims (userinfo) and Access Token Claims (decoded JWT) — alongside the
existing parsed view. Strip bearer tokens defense-in-depth in case a
non-conforming IdP places them in its userinfo response.
Resolves LIT-2838
* Update litellm/proxy/management_endpoints/ui_sso.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(sso): hoist json.dumps out of f-string for py3.10 ruff
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
