``get_llm_provider_logic.py`` instantiates ``ChatGPTConfig`` /
``GithubCopilotConfig`` and calls ``_get_openai_compatible_provider_info``
at every ``add_deployment`` cycle during proxy startup (30s tick when
``STORE_MODEL_IN_DB=True``). The old code eagerly called
``get_access_token()`` / ``get_api_key()`` through the filesystem
authenticator at that point. With no tokens on disk this falls through
to ``_login_device_code()``, which prints a device prompt to stdout and
polls the IdP for up to 15 minutes — blocking startup and (for OAuth
credentials stored in the DB) duplicating work that ``validate_environment``
is about to do correctly at request time.
Resolution is now a pure metadata pass: pick an ``api_base``, let the
``oauth:<name>`` marker pass through untouched. Actual token resolution
still happens at request time via ``resolve_authenticator`` inside
``validate_environment``, which is where we have the full
``litellm_params`` anyway.
Tests updated to assert ``get_access_token`` / ``get_api_key`` are NOT
called during resolution, and that the ``oauth:`` marker passes through.
Reported symptom from a working OAuth request to
``chatgpt.com/backend-api/codex/responses``:
APIConnectionError: ChatgptException - Unknown items in responses
API response: []
The ChatGPT / Codex backend ships the response body as SSE. Each
output item arrives in its own ``response.output_item.done`` event,
and the terminal ``response.completed`` frame carries an empty
``response.output`` array — it is effectively a "we're done" signal
rather than the carrier for the items.
The existing parser only read ``response.output`` off the completed
frame, so it handed the downstream chat-completions translator a
``ResponsesAPIResponse`` with ``output=[]``, which blew up on
``_convert_response_output_to_choices``.
Fix: accumulate items from ``response.output_item.done`` during the
loop and, when ``response.completed`` arrives with an empty output,
substitute the accumulated list. The canonical OpenAI shape (items
already populated on ``response.completed``) still works — we only
fill in when the completed frame itself is empty.
Extracted the SSE loop into ``_parse_codex_sse_response`` to keep
``transform_response_api_response`` under the PLR0915 threshold.
Added a regression test that reproduces the Codex wire shape: one
``output_item.done`` event followed by a ``response.completed`` with
``output=[]``. 174 tests pass; Black + Ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reported symptom: Test Model fails immediately with
GetLLMProvider Exception - Failed to request device code:
Client error '403 Forbidden' for url
'https://auth.openai.com/api/accounts/deviceauth/usercode'
No 15-minute hang this time — it comes back instantly.
Root cause: the ChatGPT ``DBAuthenticator`` inherited
``get_access_token`` from the filesystem ``Authenticator``, whose last
line is:
tokens = self._login_device_code()
That's the right fall-through for the CLI (no tokens on disk → start a
new login). In the proxy's DB-backed context it is catastrophic: the
server tries to initiate an unattended device-code request, OpenAI
returns 403 on the ``usercode`` endpoint (no browser to walk through
consent), and the admin sees the cryptic error above.
Copilot's ``DBAuthenticator`` already overrides ``get_access_token``
and raises ``GetAccessTokenError`` cleanly — this commit mirrors that
in the ChatGPT side:
- If no credential is loaded in ``litellm.credential_list``, raise a
401 pointing the admin at STORE_MODEL_IN_DB and the UI sign-in.
- If the token is expired and refresh fails, raise a 401 quoting the
IdP's refresh error.
- Never call ``_login_device_code`` from the proxy path.
Added regression tests that mock ``_login_device_code`` and assert it
is never called across the three failure modes above (missing
credential, happy path, refresh-error). 173 tests pass; Black + Ruff
clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reported symptom: "Test model" in the UI hangs indefinitely, a new
device code appears in the proxy logs, and the whole UI is frozen.
Root cause: the ChatGPT **chat** transformation still used the
filesystem-only ``Authenticator`` unconditionally. When the UI's
``/health/test_connection`` runs in the default ``mode="chat"``:
UI → /health/test_connection (sync wait)
→ litellm.ahealth_check(mode="chat")
→ litellm.acompletion(...)
→ ChatGPTConfig._get_openai_compatible_provider_info(api_key="oauth:X")
→ self.authenticator.get_access_token() # filesystem authenticator
→ _login_device_code() # no tokens on disk → prints a new
# device code and polls for up to 15
# minutes, blocking the request
``ChatGPTResponsesAPIConfig`` already had the ``resolve_authenticator``
dispatch (the PR's original scope), so the responses-mode path was
fine. The chat path was overlooked.
Fix:
- Apply ``resolve_authenticator`` to both call-sites in
``ChatGPTConfig`` (``_get_openai_compatible_provider_info`` and
``validate_environment``). When ``api_key`` carries the ``oauth:``
prefix, both methods now reach for the DB-backed authenticator and
the stored credential — no device-code prompt, no 15-minute hang.
- Extend ``resolve_authenticator`` to the 3-arg shape
``(api_key, litellm_params, fallback)`` so it matches the Copilot
resolver and covers the chat call-site that has ``api_key`` but not
``litellm_params``. Updated the single existing call-site in
``responses/transformation.py`` and the existing unit tests.
- Add a regression test asserting that ``ChatGPTConfig`` never calls
the filesystem authenticator when ``api_key`` is ``oauth:<name>``
(fs_auth is a MagicMock that raises on any attribute access).
170 tests pass; Black + Ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reported error after the cross-loop fix:
Tokens obtained but DB persist failed: Unable to match input value to
any allowed input type for the field. Parse errors: [...
``credential_values`` should be of any of the following types:
``JsonNullValueInput``, ``Json`` ...]
Prisma's Json columns ingest pre-serialized JSON *strings*, not raw
Python dicts. The ``/credentials`` endpoint wraps the payload with
``jsonify_object(...)`` (in ``litellm/proxy/utils.py``) which does
``json.dumps`` per nested-dict field. My persist helper was skipping
that step and passing raw dicts — Prisma's binding layer then couldn't
match them to the ``Json`` type.
Route the ``credential_values`` + ``credential_info`` dicts through
``jsonify_object`` before the upsert. Same fix in both ChatGPT and
Copilot ``db_authenticator.py``. Tests updated to assert the serialized
string form (``json.loads(kwargs[...]["credential_values"])``).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add "Sign in with ChatGPT" and "Sign in with GitHub Copilot" device-code
OAuth flows to the Add Credential modal. Tokens persist as encrypted JSON
in LiteLLM_CredentialsTable and are picked up at request time when a
model's api_key is set to "oauth:<credential_name>"; a DBAuthenticator
subclass reads from the in-memory credential cache (sync) and writes
refreshed tokens back via a fire-and-forget worker thread.
Per-row Refresh button rotates tokens on demand — ChatGPT via the IdP's
refresh_token grant, Copilot by re-deriving the short-lived API key from
the stored GitHub access token.
Also ships a litellm-chatgpt-login CLI with an optional PKCE+loopback
flow alongside the existing device-code path for local sign-in outside
the UI.
Provider-specific surface lives in new files under
litellm/llms/{chatgpt,github_copilot}/db_authenticator.py and
litellm/proxy/{chatgpt,copilot}_oauth_endpoints/. Shared-file delta is
~275 lines across 10 pre-existing files, the bulk being a pure append
at the end of networking.tsx.
Dispatch convention + operational caveats documented in the new
"ChatGPT / Copilot OAuth Credentials" section of CLAUDE.md.
Security:
- /chatgpt/oauth/* and /copilot/oauth/* endpoints require PROXY_ADMIN
(view-only admins excluded from write paths)
- session_id query params are Query(...) annotated
- session-cap check + slot reservation are atomic; reserved slot is
cleaned up on device-code failure
- PKCE loopback callback html.escape()s the IdP's error_description
before rendering
Tests: 165 cases covering DBAuthenticator read/write, config dispatch,
endpoint admin-only, 429 cap, slot cleanup, background-worker success /
auth-failure / DB-persist-failure paths, PKCE helpers + XSS regression,
CLI broad-exception + KeyboardInterrupt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When modify_params is true, Bedrock Converse setup no longer prepends or
appends the default user message if the boundary assistant turn has
prefix: true, so OpenAI-style assistant prefill reaches the API unchanged.
Made-with: Cursor
- Static Methods for Streaming Handler Function
- Remove the afile_content_streaming wrapper function. Enabled with a stream boolean in afile_content
- Cleaned up test cases after refactor
- Removed unused imports and streamlined type hints in `litellm/utils.py` and `litellm/files/main.py`.
- Moved `FileContentStreamingResult` to a new `litellm/files/types.py` for better organization.
- Updated `FileContentStreamingResponse` in `litellm/files/streaming.py` to include asynchronous close methods and improved logging capabilities.
- Enhanced tests to ensure proper closure of streaming iterators in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Introduced `afile_content_streaming` and `file_content_streaming` functions in `litellm/files/main.py` to handle asynchronous and synchronous file content streaming.
- Added `FileContentStreamingResponse` class in `litellm/files/streaming.py` to manage streaming responses with logging capabilities.
- Updated OpenAI API integration in `litellm/llms/openai/openai.py` to support new streaming methods.
- Enhanced file content retrieval in `litellm/proxy/openai_files_endpoints/files_endpoints.py` to route requests for streaming.
- Added unit tests for the new streaming functionality in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Refactored type hints and imports for better clarity and organization across modified files.
* build: migrate packaging metadata to uv
* ci: move automation and local tooling to uv
* docker: migrate image builds and runtime setup to uv
* docs: update install and deployment guidance for uv
* chore: align auxiliary scripts and tests with uv
* test: harden test_litellm isolation
* fix: keep release and health check images self-contained
* build: pin uv tooling and health check deps
* test: isolate bedrock image request formatting from suite state
* test: cover sandbox executor requirements flow
* ci: fix circleci no-op command steps
* ci: fix circleci publish workflow parsing
* fix: stabilize remaining uv migration CI checks
* ci: increase matrix test timeout headroom
* fix: restore published docker and license coverage
* fix: restore proxy runtime build parity
* fix: restore proxy extras parity and venv migrations
* ci: persist uv path across circleci steps
* fix: keep psycopg binary in default test env
* docker: preserve prisma cache across stages
* test: run local proxy checks through uv python
* build: restore runtime deps moved into ci
* build: refresh uv lock after upstream merge
* fix: restore module import in test_check_migration after merge
The conflict resolution imported only the function but the test body
references check_migration as a module throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching
- Move google-generativeai, Pillow, tenacity back to ci group (they are
lazily imported and bloat the base SDK install needlessly)
- Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant
in Docker where system Node.js is already installed via apk)
- Remove all nodejs-wheel node replacement and venv npm patching blocks
from Dockerfiles since the wheel is no longer installed
- Add --no-default-groups to CodSpeed benchmark workflow so the benchmark
environment matches the old minimal pip install footprint
- Apply standard uv two-phase Docker pattern: copy metadata first, install
deps (cached layer), then copy source and install project
- Replace CircleCI enterprise no-op with proper uv sync command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate uv.lock after removing nodejs-wheel-binaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): use cache/restore instead of cache to prevent cache poisoning
The old workflow used actions/cache/restore (read-only). The uv migration
changed it to actions/cache (read-write), which zizmor flags as a cache
poisoning risk. Restore the safer read-only variant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert
The setup-uv action enables caching by default, which zizmor flags as a
cache poisoning risk. Disable it since we already use a read-only
cache/restore step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv cache in publish workflow
Silences zizmor cache-poisoning alert. Publishing workflow runs
infrequently on protected branches so caching adds no real benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(test): remove duplicate verbose_logger mock in test_check_migration
The logger was patched twice — first via mocker.patch() then via
mocker.patch.object(autospec=True). The second call fails because
autospec cannot inspect an already-mocked attribute. Remove the
redundant first patch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): free disk space before Docker build in test-server-root-path
The Dockerfile.non_root build ran out of disk on the CI runner. Remove
Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Normalize JSON Schema type custom to object for Bedrock invoke and
_bedrock_tools_pt, ensure stable names for tools without name, and
avoid KeyError in the Anthropic messages adapter when translating
tools to OpenAI format for bedrock/converse.
Made-with: Cursor
* fix(vertex_ai): support pluggable (executable) credential_source for WIF auth (#24700)
The WIF credential dispatch in load_auth() only handled identity_pool and
aws credential types. When credential_source.executable was present (used
for Azure Managed Identity via Workload Identity Federation), it fell
through to identity_pool.Credentials which rejected it with MalformedError.
Add dispatch to google.auth.pluggable.Credentials for executable-type
credential sources, following the same pattern as the existing identity_pool
and aws helpers.
Fixes authentication for Azure Container Apps → GCP Vertex AI via WIF
with executable credential sources.
* feat(logging): add component and logger fields to JSON logs for 3rd p… (#24447)
* feat(logging): add component and logger fields to JSON logs for 3rd party filtering
* Let user-supplied extra fields win over auto-generated component/logger, tighten test assertions
* Feat - Add organization into the metrics metadata for org_id & org_alias (#24440)
* Add org_id and org_alias label names to Prometheus metric definitions
* Add user_api_key_org_alias to StandardLoggingUserAPIKeyMetadata
* Populate user_api_key_org_alias in pre-call metadata
* Pass org_id and org_alias into per-request Prometheus metric labels
* Add test for org labels on per-request Prometheus metrics
* chore: resolve test mockdata
* Address review: populate org_alias from DB view, add feature flag, use .get() for org metadata
* Add org labels to failure path and verify flag behavior in test
* Fix test: build flag-off enum_values without org fields
* Gate org labels behind feature flag in get_labels() instead of static metric lists
* Scope org label injection to metrics that carry team context, remove orphaned budget label defs, add test teardown
* Use explicit metric allowlist for org label injection instead of team heuristic
* Fix duplicate org label guard, move _org_label_metrics to class constant
* Reset custom_prometheus_metadata_labels after duplicate label assertion
* fix: emit org labels by default, remove flag, fix missing org_alias in all metadata paths
* fix: emit org labels by default, no opt-in flag required
* fix: write org_alias to metadata unconditionally in proxy_server.py
* fix: 429s from batch creation being converted to 500 (#24703)
* add us gov models (#24660)
* add us gov models
* added max tokens
* Litellm dev 04 02 2026 p1 (#25052)
* fix: replace hardcoded url
* fix: Anthropic web search cost not tracked for Chat Completions
The ModelResponse branch in response_object_includes_web_search_call()
only checked url_citation annotations and prompt_tokens_details, missing
Anthropic's server_tool_use.web_search_requests field. This caused
_handle_web_search_cost() to never fire for Anthropic Claude models.
Also routes vertex_ai/claude-* models to the Anthropic cost calculator
instead of the Gemini one, since Claude on Vertex uses the same
server_tool_use billing structure as the direct Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(anthropic): pass logging_obj to client.post for litellm_overhead_time_ms (#24071)
When LITELLM_DETAILED_TIMING=true, litellm_overhead_time_ms was null for
Anthropic because the handler did not pass logging_obj to client.post(),
so track_llm_api_timing could not set llm_api_duration_ms. Pass
logging_obj=logging_obj at all four post() call sites (make_call,
make_sync_call, acompletion, completion). Add test to ensure make_call
passes logging_obj to client.post.
Made-with: Cursor
* sap - add additional parameters for grounding
- additional parameter for grounding added for the sap provider
* sap - fix models
* (sap) add filtering, masking, translation SAP GEN AI Hub modules
* (sap) add tests and docs for new SAP modules
* (sap) add support of multiple modules config
* (sap) code refactoring
* (sap) rename file
* test(): add safeguard tests
* (sap) update tests
* (sap) update docs, solve merge conflict in transformation.py
* (sap) linter fix
* (sap) Align embedding request transformation with current API
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) mock commit
* (sap) run black formater
* (sap) add literals to models, add negative tests, fix test for tool transformation
* (sap) fix formating
* (sap) fix models
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) commit for rerun bot review
* (sap) minor improve
* (sap) fix after bot review
* (sap) lint fix
* docs(sap): update documentation
* fix(sap): change creds priority
* fix(sap): change creds priority
* fix(sap): fix sap creds unit test
* fix(sap): linter fix
* fix(sap): linter fix
* linter fix
* (sap) update logic of fetching creds, add additional tests
* (sap) clean up code
* (sap) fix after review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) add a possibility to put the service key by both variants
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) update test
* (sap) update service key resolve function
* (sap) run black formater
* (sap) fix validate credentials, add negative tests for credential fetching
* (sap) fix validate credentials, add negative tests for credential fetching
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) lint fix
* (sap) lint fix
* feat: support service_tier in gemini
* chore: add a service_tier field mapping from openai to gemini
* fix: use x-gemini-service-tier header in response
* docs: add service_tier to gemini docs
* chore: add defaut/standard mapping, and some tests
* chore: tidying up some case insensitivity
* chore: remove unnecessary guard
* fix: remove redundant test file
* fix: handle 'auto' case-insensitively
* fix: return service_tier on final steamed chunk
* chore: black
* feat: enable supports_service_tier to gemini models
* Fix get_standard_logging_metadata tests
* Fix test_get_model_info_bedrock_models
* Fix test_get_model_info_bedrock_models
* Fix remaining tests
* Fix mypy issues
* Fix tests
* Fix merge conflicts
* Fix code qa
* Fix code qa
* Fix code qa
* Fix greptile review
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Josh <36064836+J-Byron@users.noreply.github.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Alperen Kömürcü <alperen.koemuercue@sap.com>
Co-authored-by: Vasilisa Parshikova <vasilisa.parshikova@sap.com>
Co-authored-by: Lin Xu <lin.xu03@sap.com>
Co-authored-by: Mark McDonald <macd@google.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
DashScope inherits OpenAIGPTConfig which strips cache_control from
messages and tools by default. Override remove_cache_control_flag_from_messages_and_tools()
to preserve cache_control, following the same pattern used by ZAI, MiniMax, and Databricks.
Verified through 10-round multi-turn conversation tests:
- Explicit caching works correctly: cached_tokens grows each round from R4 onwards,
with cache_creation_tokens reported on first cache build.
- Implicit caching is not affected: models that rely on implicit prefix-matching caching
produce identical cached_tokens with and without this change, confirmed by comparing
results against both the reverted codebase and direct API calls bypassing litellm.
- No errors or regressions observed on any model, including those that do not support
explicit caching — the DashScope API silently ignores unrecognized cache_control fields.
Fixes#25330
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Anthropic/Claude Code use input_schema.type "custom"; Bedrock rejects it.
- Add normalize_json_schema_custom_types_to_object and use it for Invoke,
chat invoke, and _bedrock_tools_pt (Anthropic input_schema + OpenAI params).
- Coerce invalid root types to object for Converse toolSpec.
- Tests for invoke transform, converse _bedrock_tools_pt, and unit helper.
Made-with: Cursor
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry
Missing unversioned entry causes cost tracking to return $0.00 for
all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI
Claude models have both versioned and unversioned entries.
* fix(router): skip misleading tags error when no candidates (e.g. cooldown)
Return early from get_deployments_for_tag when healthy_deployments is empty so
tag-based routing does not raise no_deployments_with_tag_routing after cooldown
filters all deployments. Adds regression test.
Made-with: Cursor
* feat(oci): add embedding support and update model catalog
- Add OCIEmbeddingConfig for OCI GenAI embedding models
- Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini)
- Add 8 embedding models (Cohere embed v3.0, v4.0)
- Update documentation with embedding examples
- Update pricing for all new models
* test(oci): add unit tests for OCI embedding support
- 17 unit tests covering OCIEmbeddingConfig
- Tests for URL generation, param mapping, request/response transform
- Tests for model pricing JSON completeness
* style(oci): format with black and ruff
* fix(oci): correct embedding request body format
OCI embedText API expects inputs, truncate, and inputType at the
top level of the request body, not nested under embedTextDetails.
Fixed transformation and updated tests accordingly.
Verified with real OCI API: 3/3 embedding models working.
* docs: clarify tag routing early return and test intent
Made-with: Cursor
* fix(oci): address code review findings from Greptile
- P1: Fix signing URL mismatch with custom api_base by accepting
api_base parameter in transform_embedding_request
- P2: Remove encoding_format from supported params (OCI does not
support it, was silently dropped)
- P2: Raise ValueError for token-array inputs instead of silently
converting to string representation
- Add test for token-list rejection
* fix(mcp): add STS AssumeRole support for MCP SigV4 authentication
MCPSigV4Auth only supported static AWS credentials or the boto3 default
credential chain. Production Kubernetes environments typically authenticate
via IAM role assumption (sts:AssumeRole), which was not possible.
Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth
stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole
to obtain temporary credentials before signing requests. Explicit keys,
if also provided, are used as the source identity for the STS call;
otherwise ambient credentials (pod role, instance profile) are used.
* fix: stop logging credential values and add missing redaction patterns
Replaces raw credential values in debug/error log messages with
boolean presence checks or type names. Adds PEM block, GCP token,
JWT, SAS token, and service-account blob patterns to the redaction
filter. Fixes private_key pattern to capture full PEM blocks instead
of stopping at the first whitespace.
Addresses: Vertex AI credential JSON (including RSA private key)
being logged to stderr on health check failures.
* fix: log only field names for UserAPIKeyAuth, not full object
* style: apply black formatting to experimental_mcp_client/client.py
* style: fix black/isort formatting and mypy error in proxy_server.py
- Fix black formatting in experimental_mcp_client/client.py (done in prev commit)
- Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py
- Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py
* fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue
When test_proxy_cli.py tests run before test_check_migration.py in the same
xdist worker, litellm.proxy.db.check_migration is already in sys.modules.
Patching litellm._logging.verbose_logger has no effect on the already-bound
reference. Patch the correct target (check_migration.verbose_logger) and
import the module before patching so the order doesn't matter.
* fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature
---------
Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com>
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* Litellm ishaan april1 (#25103)
* fix(proxy): enforce upperbound key params on key/update and add custom_key_update hook
The /key/update endpoint did not enforce upperbound_key_generate_params,
allowing users to bypass configured limits (tpm_limit, rpm_limit,
max_budget, duration, budget_duration) by updating an existing key
instead of generating a new one.
Extract the upperbound enforcement logic from _common_key_generation_helper()
into a standalone _enforce_upperbound_key_params() function and call it from
both the generate and update paths. For updates, None values are skipped
(not filled with defaults) since they mean "don't change this field".
Also adds a custom_key_update config option and user_custom_key_update global,
mirroring the existing custom_key_generate pattern, so custom key validation
logic can fire during key updates as well.
* fix(proxy): invoke custom_key_update hook in bulk update path
The user_custom_key_update hook was only called in update_key_fn
(single key update) but not in _process_single_key_update (bulk
update path), allowing custom validation to be bypassed via the
/key/update/bulk endpoint. Mirror the hook invocation in both paths.
* fix(proxy): pass UpdateKeyRequest to hook in bulk path, not BulkUpdateKeyRequestItem
Move the custom_key_update hook invocation to after UpdateKeyRequest
is constructed so the hook receives the same type in both single and
bulk update paths. Previously the bulk path passed
BulkUpdateKeyRequestItem (5 fields only), which would cause
AttributeError for hooks accessing fields like tpm_limit or models.
* fix(bedrock): promote cache usage to message_delta for Claude Code (#24850)
Ensure Bedrock/Anthropic-compatible streaming exposes cache usage where Claude Code reads it by promoting message_stop usage onto message_delta and preserving usage fields in fake-streamed message_delta events.
Made-with: Cursor
* fix(search): Support self-hosted Firecrawl response format in search transform (#24866)
The `transform_search_response` method only handled Firecrawl Cloud (v2)
response format where `data` is a dict with `web`/`news` keys. Self-hosted
Firecrawl (v1) returns `data` as a flat list of result objects, causing an
`AttributeError: 'list' object has no attribute 'get'`.
Detect the response format by checking if `data` is a list (self-hosted)
or dict (cloud) and handle both cases.
Cloud format: {"data": {"web": [...], "news": [...]}}
Self-hosted: {"success": true, "data": [{"url": "...", "title": "...", ...}]}
Co-authored-by: Synergy <synergyoclaw@gmail.com>
* feat: add environment and user tracking to prompt management (#24855)
* feat: add environment and user tracking to prompt management
- Add environment (development/staging/production) and created_by columns to LiteLLM_PromptTable
- Update unique constraint to [prompt_id, version, environment]
- All CRUD endpoints support environment filtering and user tracking
- Redesigned prompt detail page with environment tabs and version history
- UI: environment filter on list page, environment selector in editor
- 8 new tests for environment and user tracking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Black formatting and add environments to PromptInfoResponse TypeScript type
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address Greptile review findings
- P1: delete_prompt scopes in-memory cleanup to environment when provided
- P2: dotprompt_content parsed directly regardless of environment flag
- P2: use distinct for environments query
- P2: fix double-fetch on initial mount in prompt_info.tsx
- fix: remove unsupported select kwarg from find_many
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address remaining Greptile review comments
- Remove unused useCallback import (index.tsx)
- Remove unused ENV_COLORS variable (prompt_info.tsx)
- P1: in-memory fallback in get_prompt_versions now respects environment filter
- P1: reset selectedEnv when promptId changes to avoid stale state
- Cyclic imports are pre-existing pattern, not introduced by this PR
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: scope patch_prompt to environment using primary key
- Add environment query param to patch_prompt endpoint
- Look up target row by composite key (prompt_id + version + environment)
- Update by primary key (id) to target exactly one row
- Fixes Greptile finding: patch with multiple environments no longer ambiguous
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use actual start_time for failed request spend logs (#24906)
async_post_call_failure_hook set both start_time and end_time to
datetime.now(), making all failed requests show duration=0. Use the
actual start_time from litellm_logging_obj instead, so spend logs
reflect the real request duration on timeout and other failures.
Fixes#24888
* feat(bedrock): add nova canvas image edit support (#24869)
* feat(bedrock): add nova canvas image edit support
* fix(bedrock): support PathLike inputs for nova image edit
* chore: sync schema.prisma copies from root
* fix(mypy): correct type-ignore code for delta_usage arg-type
* fix(mypy): cast status_code to str, suppress intentional str yield
* fix(lint): extract _create_content_block_chunks to fix PLR0915
* fix(lint): extract helpers to fix PLR0915 in prompt endpoints
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: redhelix <amin.lalji@gmail.com>
Co-authored-by: Synergy <synergyoclaw@gmail.com>
Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: madhu19991 <madhu@thunkai.com>
Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(test): update model armor streaming test to handle string or int error code
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: redhelix <amin.lalji@gmail.com>
Co-authored-by: Synergy <synergyoclaw@gmail.com>
Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: madhu19991 <madhu@thunkai.com>
Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(proxy): enforce key-level model allowlist for custom auth
custom_auth_run_common_checks only runs common_checks (team/user/project model checks).
Custom auth now also enforces key-level model restrictions via can_key_call_model.
Move the custom-auth key-access regression tests to test_user_api_key_auth.py and keep test_custom_auth_end_user_budget.py focused on end-user budget behavior.
Made-with: Cursor
* fix(proxy): gate custom-auth key model checks behind opt-in
Keep key-level model allowlist enforcement in custom auth behind `custom_auth_run_common_checks` to preserve backwards compatibility, and update tests to verify default non-enforcement and opt-in enforcement behavior.
Made-with: Cursor
* test(proxy): isolate custom auth default check from shared settings state
Patch `proxy_server.general_settings` to an empty dict in the default custom-auth key-access test so it remains deterministic under shared module state.
Made-with: Cursor
* test(proxy): strengthen custom auth post-check assertions
Tighten custom auth regression tests by asserting exact can_key_call_model args and remove an unused common_checks mock from the default behavior path.
Made-with: Cursor
* fix(agentcore): parse A2A JSON-RPC responses in AgentCore provider
* fix(prompt-templates): ensure_alternating_roles handles tool-call chains
* feat(auth): add JWT claim routing overrides for OAuth2 validation
Made-with: Cursor
* docs(auth): document JWT-to-OAuth2 routing overrides
Add generic docs for running JWT and OAuth2 together, including routing_overrides YAML examples and list-based selector behavior for iss/client_id/aud.
Made-with: Cursor
---------
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* update bedrock models in tests
* updated more tests and model_prices_and_context_window
* fix model id and pricing
* replace more sonnet models
* update tests
* git push
* update pricing
* flaky total cost
* monkey patch
* relax the cost change
* fix and revert some changes
* revert the pricing
* chore: move cost/pricing changes to bedrock-cost-fixes branch
* chore: split Bedrock file-api beta stripping to separate branch
Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch;
see litellm_bedrock_invoke_strip_file_api_betas for that fix.
Made-with: Cursor
Main rewrote the same tests we moved. Resolution: keep the tests only
in the unit test directory, adopting main's improved patterns (AsyncMock,
assert_called_once, stronger content assertions on streaming).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move test_chat_completion_snowflake and test_chat_completion_snowflake_stream
from tests/llm_translation/ to tests/test_litellm/llms/snowflake/chat/ so
they run as part of `make test-unit` without requiring API credentials.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
