- Extend responses_api_bridge_check when reasoning_effort + summary aliases
(including nested extra_body) without tools
- Merge summary into reasoning_effort for responses bridge; helpers in utils
- Strip summary aliases in GPT-5 chat mapping when not bridged
- Tests for bridge + merge behavior
Co-authored-by: Cursor <cursoragent@cursor.com>
Stripping body-supplied tags in apply_client_tag_policy_pre_auth silently
disabled per-tag budget enforcement for non-opted-in keys — pre-PR
behavior was that those tags reached _tag_max_budget_check inside
common_checks. The post-auth strip in add_litellm_data_to_request
continues to remove unauthorized tags before they leave the proxy.
Also moves the LiteLLMProxyRequestSetup import to module-level (no
circular dep with auth/user_api_key_auth.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The x-litellm-tags header was merged into request metadata only after the
auth chain completed, so _tag_max_budget_check (which reads tags from the
request body) silently failed open for header-tagged requests — spend
accumulated past max_budget without any 400 budget_exceeded response.
Move the client-tag policy (strip-or-merge gated on allow_client_tags) to
run before common_checks so header tags are visible to budget enforcement.
The post-auth strip+merge in add_litellm_data_to_request stays as
defense-in-depth; the new pre-auth helper is idempotent with it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mcp): forward extra_headers for OpenAPI MCP tools
OpenAPI-generated tools only applied static closure headers and BYOK
Authorization via ContextVar. Copy MCPServer.extra_headers from the
incoming MCP request into _request_extra_headers (set in server.py before
local tool dispatch), merge in openapi_to_mcp_generator via a small helper.
OAuth2 M2M: do not forward caller Authorization from raw_headers (same rule
as _prepare_mcp_server_headers for managed MCP).
Adds TestRequestExtraHeaders and clarifies mcp_server_manager registration
comment.
Fixes#26794
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): access has_client_credentials on MCPServer directly
Greptile: getattr default was redundant; property exists on MCPServer and
mcp_server is non-None inside the extra_headers forwarding block.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The previous detection treated any model with input_cost_per_image
or output_cost_per_image as image generation. Several chat and
embedding models carry those fields to price multimodal vision input,
not generated images:
- gemini-3.1-pro-preview (mode=chat) has output_cost_per_image=0.00012
alongside input/output token pricing.
- azure/gpt-realtime-* (mode=chat) has input_cost_per_image=5e-6.
- amazon.titan-embed-image-v1 (mode=embedding) has
input_cost_per_image=6e-5.
For these models the image-gen branch fired first and reserved a
fraction of a cent per request, short-circuiting the token-priced
path entirely. Long Gemini chats reserved 1 × $0.00012 instead of
the true token cost.
Gate strictly on mode in {"image_generation", "image_edit"}. All 197
real image_generation entries and all 31 image_edit entries
(Flux Kontext, Stability inpaint/outpaint, etc.) carry the right mode,
so the field-presence fallback was unnecessary.
Adds regression tests for the chat-model-with-image-cost-field case
and for image_edit reservation.
Image-generation routes (dall-e-3, flux, etc.) have no per-token output
cost so they fell through to the no-reservation read-time-only path.
Concurrent image requests against a depleted budget could all pass
common_checks (counter exactly at max_budget passes the strict-`>`
gate) and reach the provider before reconciliation caught up.
Add per-image reservation in _estimate_request_max_cost_for_model:
when the model has a per-image cost field, reserve `n × cost_per_image`
upfront. The atomic counter increment serializes concurrent admissions,
so the second request sees the post-first-reservation counter and
raises BudgetExceededError instead of silently leaking through.
Both `output_cost_per_image` and `input_cost_per_image` are honored —
naming is inconsistent across providers (OpenAI dall-e-3 uses
input_cost_per_image, aiml/dall-e-3 uses output_cost_per_image for
the same per-generated-image price).
Per-pixel pricing (DALL-E 2 size variants) and TTS/STT routes still
fall through to read-time enforcement; those are follow-ups.
- Introduce RoutingPrismaWrapper that transparently routes read operations (find_*, count, group_by, query_raw, query_first) to a reader endpoint while writes remain on the writer, enabling Aurora-style reader/writer endpoint splits
- Add IAMEndpoint dataclass and parse_iam_endpoint_from_url() to capture static connection fields from a reader URL so only the IAM token needs to rotate, avoiding the need for separate DATABASE_HOST_READ_REPLICA/etc. env vars
- Enhance PrismaWrapper with per-instance knobs (db_url_env_var, iam_endpoint, recreate_uses_datasource, log_prefix) so writer and reader wrappers are independent: the reader writes its fresh URL to DATABASE_URL_READ_REPLICA and passes datasource override to Prisma since Prisma only auto-reads DATABASE_URL
- Fix deadlock in PrismaWrapper.__getattr__: when called from inside a running event loop, schedule the token refresh as a background task instead of blocking with run_coroutine_threadsafe + future.result(), which would deadlock the loop thread waiting for a coroutine that needs the loop to run
- Fix botocore crash when DATABASE_PORT is unset by defaulting to "5432" in both proxy_cli.py and PrismaWrapper.get_rds_iam_token(); passing None caused botocore to embed the literal string "None" in the presigned URL
- Implement graceful reader degradation: reader connect/recreate failures are non-fatal; wrapper sets _reader_unavailable=True and silently routes reads to the writer to keep the proxy serving traffic during transient reader outages
- Add PrismaClient.writer_db property so the reconnect smoke-test always validates the writer engine specifically; query_raw on the routing wrapper would route to the reader and not verify the newly-recreated writer
- Expose DATABASE_URL_READ_REPLICA in Helm chart (values.yaml + deployment.yaml) via both plain value and secret key reference, and document the field in docker-compose.yml
- Add 887-line test suite covering routing logic, IAM token refresh paths, reader degradation scenarios, datasource override behavior, and the deadlock regression
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
reserve_budget_for_request fell back to reserving the entire remaining
team/key/user headroom whenever a request omitted max_tokens, which
pinned the spend counter at max_budget for the duration of the
in-flight request and false-positive-blocked every concurrent or
back-to-back request until the success callback reconciled. Surfaced
as an integration-test team being budget-blocked at its $2000 cap
while DB spend was $0.144.
Switch the missing-max_tokens path to a fixed default of 16384 output
tokens (mirrors parallel_request_limiter_v3's DEFAULT_MAX_TOKENS_ESTIMATE
precedent), and clamp explicit max_tokens at the model's
max_output_tokens for reservation accounting only. The outbound request
body is unchanged, so providers see whatever the caller actually sent;
only the local integer used to compute reservation cost is bounded.
This also prevents a hostile max_tokens=999999999 from inflating one
request's reservation up to the entire team headroom.
For Opus 4.7 (output $25/M, max_output 128K) on a $2000 budget the
worst-case per-request reservation drops from "everything left" to
$3.20, raising admittable concurrency from 1 to ~625.
Operators upgrading past 35bbca60b0 (which made /metrics auth
default-on) see "Malformed API Key passed in. Ensure Key has 'Bearer '
prefix." with no hint that
litellm_settings.require_auth_for_metrics_endpoint: false restores the
previous unauthenticated behavior. Append that discovery hint to the
existing 401 body so a Prometheus scraper that breaks after upgrade
has a clear migration path. No behavior change.
* feat(sso): show full IdP claims in /sso/debug/callback
The debug callback only displayed the proxy-parsed OpenID summary, so
customers couldn't verify what custom claims (team_id, team_alias, roles,
etc.) the IdP was actually returning. Render two new sections — Raw
Claims (userinfo) and Access Token Claims (decoded JWT) — alongside the
existing parsed view. Strip bearer tokens defense-in-depth in case a
non-conforming IdP places them in its userinfo response.
Resolves LIT-2838
* Update litellm/proxy/management_endpoints/ui_sso.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(sso): hoist json.dumps out of f-string for py3.10 ruff
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Mirrors the system-message skip in PR #25481 for tool-role messages.
Adds a global litellm.skip_tool_message_in_guardrail flag and a
per-guardrail litellm_params.skip_tool_message_in_guardrail override,
applied in the OpenAI and Anthropic chat translation handlers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Google updated their Interactions OpenAPI spec
(https://ai.google.dev/static/api/interactions.openapi.json), removing
the readOnly 'outputs' property from CreateModelInteractionParams in
favor of 'steps' (a polymorphic transcript array). The compliance test
fetches the live spec, so it began failing on every PR once the spec
flipped over. Update the asserted output-field list to match.
Note: this only re-aligns the spec-shape assertion. Our SDK response
types (litellm/types/interactions/generated.py) still expose 'outputs'
and need to be regenerated separately to add 'steps'/Step variants and
decide on a back-compat path for callers reading .outputs.
Strip VCR wiring from the batches test conftest. Drops:
- import of `_vcr_conftest_common` helpers
- the `vcr_config` fixture, `pytest_recording_configure`,
`_vcr_outcome_gate`, `pytest_runtest_makereport`
- the `apply_vcr_auto_marker_to_items` call in
`pytest_collection_modifyitems`
- `VerboseReporterState` / its `pytest_configure` /
`pytest_runtest_logreport` hooks (purely VCR-verdict plumbing)
Why: every test in this directory creates ephemeral OpenAI / Bedrock /
vLLM resources whose IDs change per run (file-XXX, batch-XXX,
ft-XXX, ...). VCR's path/query/body matchers don't match across runs,
so `record_mode="new_episodes"` was silently passing through to the
live API and recording many new cassette entries every run. Cassette
bloat without replay benefit.
Behaviour after this change is identical to running the directory
without `CASSETTE_REDIS_URL` set: tests that have keys hit live APIs,
tests that don't continue to skip via their existing skipif markers.
Conftest now keeps only path setup and the session-scoped `event_loop`
fixture.
OpenAI announced gpt-3.5-turbo-0125 (and fine-tuning of gpt-3.5-turbo
in general) for shutdown on 2026-10-23, with the announcement landing
2026-04-22. The hard-fail date is ~5 months out, but timing fits the
recent uptick in this test flaking and OpenAI may already be running
the deprecated model's pipeline with deprioritized infra.
Bump to gpt-4o-mini-2024-07-18 — currently supported for fine-tuning,
no announced shutdown. Updates the live test plus the mocked test for
consistency. Belt-and-suspenders with the existing propagation-retry
helper.
Previous fix polled `litellm.afile_retrieve` for `status == "processed"`
before calling the fine-tuning endpoint. That doesn't actually solve
the race:
- OpenAI's `FileObject.status` field is deprecated per the SDK type and
not authoritative — it can read "processed" before the file is usable.
- The retrieve and fine-tuning endpoints don't share a consistency
model, so retrieve succeeding tells you nothing about FT visibility.
Replace with a retry around the actual `acreate_fine_tuning_job` call
that catches the OpenAI 400 `'file-... does not exist'` and backs off
exponentially (1s → cap 8s, 12 attempts, ~70s total budget). The
operation succeeding is the only reliable signal that propagation
finished.
OpenAI file uploads are eventually consistent — a freshly uploaded file
may briefly 404 from `retrieve` and is rejected by the fine-tuning
endpoint with `'file-... does not exist'` until processing finishes.
The async fine-tuning test called `acreate_fine_tuning_job` immediately
after `acreate_file` and flaked on this race.
Add a polling helper that waits up to ~30s for `status=processed` (and
short-circuits on `error`), called between upload and FT job creation.
Mirrors the same propagation lag covered by the `await asyncio.sleep(1)`
in the sister batches test, but more robust against longer delays.
OpenAI deprecated the gpt-4o-realtime-preview-2024-10-01 snapshot,
which caused these E2E tests to fail consistently in CI. Bump to the
unversioned gpt-4o-realtime-preview alias to match the sibling
test_openai_realtime_simple.py and stay current as OpenAI rolls the
alias forward.
