- Route multipart forwarding on forward_multipart instead of empty _parsed_body
so litellm_logging_obj no longer forces json= for file uploads.
- Remove custom_body from pass-through endpoint signatures; FastAPI treated it
as a JSON body and rejected multipart before the handler ran. Bedrock passes
JSON via request.state (LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY).
- Use build_request + send(stream=True) for streaming multipart; httpx 0.28
AsyncClient.request does not accept stream=.
- Add regression test for non-empty _parsed_body multipart path; update Bedrock
custom-body test and query-params test for forward_multipart.
Made-with: Cursor
Use Flex, Typography.Paragraph (with copyable), and Typography.Text
instead of raw divs + code block + CopyToClipboard wrapper. Drops the
direct react-copy-to-clipboard dependency in this component in favor
of antd's native copyable support.
Also fixes two test issues surfaced when running the e2e locally:
- RegenerateKeyModal.test.tsx no longer mocks react-copy-to-clipboard
(the component no longer imports it), removing the CJS require()
inside an ESM mock factory flagged by Greptile.
- keys.spec.ts scopes the Regenerate and Copy lookups to the modal.
The Regenerate button has an icon whose aria-label ("sync") is
concatenated into the button's accessible name, so an exact-match
lookup on "Regenerate" failed; and the new Paragraph copyable
renders a generic "Copy" button that collided with the other
copyable fields on the key info view.
Non-admins previously skipped RBAC when prisma_client was None but could
still read payloads from custom loggers. Return 403 unless admin view.
Add test_ui_view_request_response_forbids_non_admin_without_db.
Made-with: Cursor
- Drop module-level common_utils import; import team helpers inside callers.
- Inline admin-view role check in _is_admin_view_safe to break import cycle.
- Require non-null row.user before treating spend log as owned by the key
(fixes None==None bypass for service keys).
- Document deferred proxy_server imports in _get_permitted_team_ids_for_spend_logs.
- Update tests (common_utils patches, regression test, ruff cleanups).
Made-with: Cursor
- Introduced `afile_content_streaming` and `file_content_streaming` functions in `litellm/files/main.py` to handle asynchronous and synchronous file content streaming.
- Added `FileContentStreamingResponse` class in `litellm/files/streaming.py` to manage streaming responses with logging capabilities.
- Updated OpenAI API integration in `litellm/llms/openai/openai.py` to support new streaming methods.
- Enhanced file content retrieval in `litellm/proxy/openai_files_endpoints/files_endpoints.py` to route requests for streaming.
- Added unit tests for the new streaming functionality in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Refactored type hints and imports for better clarity and organization across modified files.
The cookbook example pinned litellm==1.61.15 which has 3 known
vulnerabilities (CVE-2026-35029, CVE-2026-35030, and a password
hash exposure issue), all patched in 1.83.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
* build: migrate packaging metadata to uv
* ci: move automation and local tooling to uv
* docker: migrate image builds and runtime setup to uv
* docs: update install and deployment guidance for uv
* chore: align auxiliary scripts and tests with uv
* test: harden test_litellm isolation
* fix: keep release and health check images self-contained
* build: pin uv tooling and health check deps
* test: isolate bedrock image request formatting from suite state
* test: cover sandbox executor requirements flow
* ci: fix circleci no-op command steps
* ci: fix circleci publish workflow parsing
* fix: stabilize remaining uv migration CI checks
* ci: increase matrix test timeout headroom
* fix: restore published docker and license coverage
* fix: restore proxy runtime build parity
* fix: restore proxy extras parity and venv migrations
* ci: persist uv path across circleci steps
* fix: keep psycopg binary in default test env
* docker: preserve prisma cache across stages
* test: run local proxy checks through uv python
* build: restore runtime deps moved into ci
* build: refresh uv lock after upstream merge
* fix: restore module import in test_check_migration after merge
The conflict resolution imported only the function but the test body
references check_migration as a module throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching
- Move google-generativeai, Pillow, tenacity back to ci group (they are
lazily imported and bloat the base SDK install needlessly)
- Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant
in Docker where system Node.js is already installed via apk)
- Remove all nodejs-wheel node replacement and venv npm patching blocks
from Dockerfiles since the wheel is no longer installed
- Add --no-default-groups to CodSpeed benchmark workflow so the benchmark
environment matches the old minimal pip install footprint
- Apply standard uv two-phase Docker pattern: copy metadata first, install
deps (cached layer), then copy source and install project
- Replace CircleCI enterprise no-op with proper uv sync command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate uv.lock after removing nodejs-wheel-binaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): use cache/restore instead of cache to prevent cache poisoning
The old workflow used actions/cache/restore (read-only). The uv migration
changed it to actions/cache (read-write), which zizmor flags as a cache
poisoning risk. Restore the safer read-only variant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert
The setup-uv action enables caching by default, which zizmor flags as a
cache poisoning risk. Disable it since we already use a read-only
cache/restore step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv cache in publish workflow
Silences zizmor cache-poisoning alert. Publishing workflow runs
infrequently on protected branches so caching adds no real benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(test): remove duplicate verbose_logger mock in test_check_migration
The logger was patched twice — first via mocker.patch() then via
mocker.patch.object(autospec=True). The second call fails because
autospec cannot inspect an already-mocked attribute. Remove the
redundant first patch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): free disk space before Docker build in test-server-root-path
The Dockerfile.non_root build ran out of disk on the CI runner. Remove
Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add PromptGuard guardrail integration
Add PromptGuard as a first-class guardrail vendor in LiteLLM's proxy,
supporting prompt injection detection, PII redaction, topic filtering,
entity blocklists, and hallucination detection via PromptGuard's
/api/v1/guard API endpoint.
Backend:
- Add PROMPTGUARD to SupportedGuardrailIntegrations enum
- Implement PromptGuardGuardrail (CustomGuardrail subclass) with
apply_guardrail handling allow/block/redact decisions
- Add Pydantic config model with api_key, api_base, ui_friendly_name
- Auto-discovered via guardrail_hooks/promptguard/__init__.py registries
Frontend:
- Add PromptGuard partner card to Guardrail Garden with eval scores
- Add preset configuration for quick setup
- Add logo to guardrailLogoMap
Tests:
- 30 unit tests covering configuration, allow/block/redact actions,
request payload construction, error handling, config model, and
registry wiring
* Fix redact path and init ordering per review feedback
- P1: Update structured_messages (not just texts) when PromptGuard
returns a redact decision, so PII redaction is effective for the
primary LLM message path
- P2: Validate credentials before allocating the HTTPX client so
resources aren't acquired if PromptGuardMissingCredentials is raised
- Add tests for structured_messages redaction and texts-only redaction
* Harden PromptGuard integration: fail-open, event hooks, images, docs
- Add block_on_error config (default fail-closed, configurable fail-open)
- Declare supported_event_hooks (pre_call, post_call) like other vendors
- Forward images from GenericGuardrailAPIInputs to PromptGuard API
- Wrap API call in try/except for resilient error handling
- Add comprehensive documentation page with config examples
- Register docs page in sidebar alongside other guardrail providers
- Expand test suite from 32 to 40 tests covering new functionality
* Fix dict[str, Any] -> Dict[str, Any] for Python 3.8 compat
* Address remaining Greptile feedback: timeout, redact guard
- Add explicit 10s timeout to async_handler.post() to prevent
indefinite hangs when PromptGuard API is unresponsive
- Guard redact path: only update inputs["texts"] when the key
was originally present, avoiding phantom key injection
- Add test: redact with structured_messages only does not create
texts key (41 tests total)
* Fix CI lint: black formatting, add PromptGuardConfigModel to LitellmParams
- Reformat promptguard.py to match CI black version (parenthesization)
- Add PromptGuardConfigModel as base class of LitellmParams for proper
Pydantic schema validation, consistent with all other guardrail vendors
- Use litellm_params.block_on_error directly (now a typed field)
* Address Greptile review: redact path, null decision, error context
- P1: Filter _extract_texts_from_messages to user-role messages only,
preventing system/assistant content from being injected into texts
- P1: Strengthen test_redact_updates_structured_messages assertion from
weak `in` check to strict equality, catching the injection bug
- P2: Use `result.get("decision") or "allow"` to handle explicit null
decision values (not just absent keys)
- P2: Wrap bare exception re-raise in GuardrailRaisedException so the
caller knows which guardrail failed (block_on_error=True path)
- P2: Add static Promptguard entry in guardrail_provider_map so the
preset works before populateGuardrailProviderMap is called
- Add test for explicit null decision treated as allow
* Fix black formatting: collapse f-string in error message
The deployment hook now converts stream=True→False in wrapper_async's
scope so the streaming early-return path is skipped and logging executes.
logging_obj.stream is synced after the hook, and the original stream
intent is recovered for the short-circuit path.
Made-with: Cursor
Adopt a React Query mutation for policy attachment deletion and add a pending-state test on the policies index panel. This removes local delete-loading state and keeps modal loading tied to mutation status.
Made-with: Cursor
Replace Tremor components in the regenerate key modal with Ant Design
equivalents and move the component to a new PascalCase file. The form
layout now uses Row/Col to place Max Budget, TPM Limit, and RPM Limit
on one row and Expire Key with Grace Period on another, reducing the
vertical footprint. The success view shows an Alert banner, the key
alias as secondary context, and the regenerated key in a monospace
block with an inline primary Copy button.
Also adds unit tests for the new component and updates the existing
Playwright spec to match the new banner and button text.
enable_tag_filtering: false (the default) passed the hasValues guard,
causing every team save to send router_settings even for teams that
never had them configured. This could silently overwrite backend-set
values. Now exclude false and empty arrays from the check.
Remove the silent try/catch from setSecureItem so OAuth hooks can
surface actionable "enable storage" guidance instead of a cryptic
"state lost" error after the round-trip. Add a local try/catch in
ChatUI where the storage write is non-critical.
Normalize JSON Schema type custom to object for Bedrock invoke and
_bedrock_tools_pt, ensure stable names for tools without name, and
avoid KeyError in the Anthropic messages adapter when translating
tools to OpenAI format for bedrock/converse.
Made-with: Cursor
Remove the onChange/value feedback loop between TeamInfo and
RouterSettingsAccordion. The parent passes initial value once and reads
current values at save time via ref.getValue(). This eliminates the
circular data flow that caused stale state and fallback tab resets.
When a fallback model was selected, the onChange effect fired, parent
re-rendered with new value prop, and the useEffect re-initialized the
groups with new IDs from fallbacksToGroups. This caused the active
fallback group tab to reset to Group 1. Now internal updates always
skip re-initialization regardless of value key changes.
The uncontrolled TabGroup reset to the first tab whenever the component
re-rendered due to onChange state updates. Using controlled index/onIndexChange
keeps the user on the current tab during fallback selection.
When a user clears a numeric field in the RouterSettingsAccordion, the empty
input should result in null, not fall through to the previous value from state.
The RouterSettingsAccordion uses uncontrolled DOM inputs for numeric fields.
The onChange effect only fires when formValue or fallbacks change, not when
the user types into uncontrolled inputs. Using the ref's getValue() method
at save time ensures we read fresh values from the DOM.
Add RouterSettingsAccordion to the team edit form and a read-only
display of router settings (routing strategy, retries, fallbacks,
cooldown, timeout, tag filtering) in the Settings tab.
Add Baseten Model API pricing entries for Nemotron, GLM, Kimi, GPT OSS, and DeepSeek models with validated model slugs. Include a focused regression test to assert provider and per-token pricing values.
Made-with: Cursor
* fix(vertex_ai): support pluggable (executable) credential_source for WIF auth (#24700)
The WIF credential dispatch in load_auth() only handled identity_pool and
aws credential types. When credential_source.executable was present (used
for Azure Managed Identity via Workload Identity Federation), it fell
through to identity_pool.Credentials which rejected it with MalformedError.
Add dispatch to google.auth.pluggable.Credentials for executable-type
credential sources, following the same pattern as the existing identity_pool
and aws helpers.
Fixes authentication for Azure Container Apps → GCP Vertex AI via WIF
with executable credential sources.
* feat(logging): add component and logger fields to JSON logs for 3rd p… (#24447)
* feat(logging): add component and logger fields to JSON logs for 3rd party filtering
* Let user-supplied extra fields win over auto-generated component/logger, tighten test assertions
* Feat - Add organization into the metrics metadata for org_id & org_alias (#24440)
* Add org_id and org_alias label names to Prometheus metric definitions
* Add user_api_key_org_alias to StandardLoggingUserAPIKeyMetadata
* Populate user_api_key_org_alias in pre-call metadata
* Pass org_id and org_alias into per-request Prometheus metric labels
* Add test for org labels on per-request Prometheus metrics
* chore: resolve test mockdata
* Address review: populate org_alias from DB view, add feature flag, use .get() for org metadata
* Add org labels to failure path and verify flag behavior in test
* Fix test: build flag-off enum_values without org fields
* Gate org labels behind feature flag in get_labels() instead of static metric lists
* Scope org label injection to metrics that carry team context, remove orphaned budget label defs, add test teardown
* Use explicit metric allowlist for org label injection instead of team heuristic
* Fix duplicate org label guard, move _org_label_metrics to class constant
* Reset custom_prometheus_metadata_labels after duplicate label assertion
* fix: emit org labels by default, remove flag, fix missing org_alias in all metadata paths
* fix: emit org labels by default, no opt-in flag required
* fix: write org_alias to metadata unconditionally in proxy_server.py
* fix: 429s from batch creation being converted to 500 (#24703)
* add us gov models (#24660)
* add us gov models
* added max tokens
* Litellm dev 04 02 2026 p1 (#25052)
* fix: replace hardcoded url
* fix: Anthropic web search cost not tracked for Chat Completions
The ModelResponse branch in response_object_includes_web_search_call()
only checked url_citation annotations and prompt_tokens_details, missing
Anthropic's server_tool_use.web_search_requests field. This caused
_handle_web_search_cost() to never fire for Anthropic Claude models.
Also routes vertex_ai/claude-* models to the Anthropic cost calculator
instead of the Gemini one, since Claude on Vertex uses the same
server_tool_use billing structure as the direct Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(anthropic): pass logging_obj to client.post for litellm_overhead_time_ms (#24071)
When LITELLM_DETAILED_TIMING=true, litellm_overhead_time_ms was null for
Anthropic because the handler did not pass logging_obj to client.post(),
so track_llm_api_timing could not set llm_api_duration_ms. Pass
logging_obj=logging_obj at all four post() call sites (make_call,
make_sync_call, acompletion, completion). Add test to ensure make_call
passes logging_obj to client.post.
Made-with: Cursor
* sap - add additional parameters for grounding
- additional parameter for grounding added for the sap provider
* sap - fix models
* (sap) add filtering, masking, translation SAP GEN AI Hub modules
* (sap) add tests and docs for new SAP modules
* (sap) add support of multiple modules config
* (sap) code refactoring
* (sap) rename file
* test(): add safeguard tests
* (sap) update tests
* (sap) update docs, solve merge conflict in transformation.py
* (sap) linter fix
* (sap) Align embedding request transformation with current API
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) mock commit
* (sap) run black formater
* (sap) add literals to models, add negative tests, fix test for tool transformation
* (sap) fix formating
* (sap) fix models
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) commit for rerun bot review
* (sap) minor improve
* (sap) fix after bot review
* (sap) lint fix
* docs(sap): update documentation
* fix(sap): change creds priority
* fix(sap): change creds priority
* fix(sap): fix sap creds unit test
* fix(sap): linter fix
* fix(sap): linter fix
* linter fix
* (sap) update logic of fetching creds, add additional tests
* (sap) clean up code
* (sap) fix after review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) add a possibility to put the service key by both variants
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) update test
* (sap) update service key resolve function
* (sap) run black formater
* (sap) fix validate credentials, add negative tests for credential fetching
* (sap) fix validate credentials, add negative tests for credential fetching
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) lint fix
* (sap) lint fix
* feat: support service_tier in gemini
* chore: add a service_tier field mapping from openai to gemini
* fix: use x-gemini-service-tier header in response
* docs: add service_tier to gemini docs
* chore: add defaut/standard mapping, and some tests
* chore: tidying up some case insensitivity
* chore: remove unnecessary guard
* fix: remove redundant test file
* fix: handle 'auto' case-insensitively
* fix: return service_tier on final steamed chunk
* chore: black
* feat: enable supports_service_tier to gemini models
* Fix get_standard_logging_metadata tests
* Fix test_get_model_info_bedrock_models
* Fix test_get_model_info_bedrock_models
* Fix remaining tests
* Fix mypy issues
* Fix tests
* Fix merge conflicts
* Fix code qa
* Fix code qa
* Fix code qa
* Fix greptile review
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Josh <36064836+J-Byron@users.noreply.github.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Alperen Kömürcü <alperen.koemuercue@sap.com>
Co-authored-by: Vasilisa Parshikova <vasilisa.parshikova@sap.com>
Co-authored-by: Lin Xu <lin.xu03@sap.com>
Co-authored-by: Mark McDonald <macd@google.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: expose reasoning effort fields in get_model_info and add together_ai/gpt-oss-120b
- litellm/utils.py: pass supports_none_reasoning_effort and
supports_xhigh_reasoning_effort through _get_model_info_helper so
get_model_info() returns them (previously silently dropped). Fixes#25096.
- model_prices_and_context_window.json: add together_ai/openai/gpt-oss-120b
with supports_reasoning: true so reasoning_effort is accepted for this
model without requiring drop_params. Fixes#25132.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: consolidate duplicate together_ai/openai/gpt-oss-120b entry and sync backup file
* fix: link commit to GitHub account for CLA verification
---------
Co-authored-by: Austin Varga <austin@knowmi.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
DashScope inherits OpenAIGPTConfig which strips cache_control from
messages and tools by default. Override remove_cache_control_flag_from_messages_and_tools()
to preserve cache_control, following the same pattern used by ZAI, MiniMax, and Databricks.
Verified through 10-round multi-turn conversation tests:
- Explicit caching works correctly: cached_tokens grows each round from R4 onwards,
with cache_creation_tokens reported on first cache build.
- Implicit caching is not affected: models that rely on implicit prefix-matching caching
produce identical cached_tokens with and without this change, confirmed by comparing
results against both the reverted codebase and direct API calls bypassing litellm.
- No errors or regressions observed on any model, including those that do not support
explicit caching — the DashScope API silently ignores unrecognized cache_control fields.
Fixes#25330
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove leading space from license public_key.pem
PEM must begin with -----BEGIN; a leading ASCII space breaks
cryptography.load_pem_public_key on older cryptography (e.g. 41.x),
causing OpenSSL no start line / deserialize errors.
Made-with: Cursor
* test: assert license public_key.pem loads as valid PEM
Regression guard for leading whitespace before -----BEGIN, which breaks
load_pem_public_key on older cryptography (e.g. 41.x).
Made-with: Cursor
Fixes 'LLM Provider NOT provided' errors when models are configured with
custom_llm_provider but model names lack provider prefix (e.g., 'gpt-4.1-mini'
instead of 'azure/gpt-4.1-mini').
Changes:
- Router now passes deployment's custom_llm_provider to get_llm_provider()
- Fixes 6 code paths: file creation, file content, batch operations, vector store
- Adds regression tests for file creation and file content operations
Made-with: Cursor
* fix(vertex_ai): normalize Gemini finish_reason enum through map_finish_reason in streaming handler
In the legacy vertex_ai SDK streaming path, the raw Gemini finish_reason enum name (e.g. "STOP", "MAX_TOKENS") was stored directly into self.received_finish_reason without being mapped to OpenAI-compatible values. The finish_reason_handler then compared against lowercase "stop", causing the case mismatch to prevent the tool_call override from ever firing. This fix applies map_finish_reason() so all Gemini enum names are normalized before storage.Refactor finish reason handling to use map_finish_reason function.
* refactor: use module-level map_finish_reason import; drop redundant inline import
map_finish_reason is already imported at module scope (line 49) via `from .core_helpers import map_finish_reason, process_response_headers`. The inline import added in the previous commit was redundant. Addressed Greptile review feedback.Removed unnecessary import of map_finish_reason from core_helpers.
* test: add unit tests for Gemini legacy vertex finish_reason normalisation
Added tests to ensure finish_reason normalization for Gemini legacy vertex tool calls and stop reasons.
Anthropic/Claude Code use input_schema.type "custom"; Bedrock rejects it.
- Add normalize_json_schema_custom_types_to_object and use it for Invoke,
chat invoke, and _bedrock_tools_pt (Anthropic input_schema + OpenAI params).
- Coerce invalid root types to object for Converse toolSpec.
- Tests for invoke transform, converse _bedrock_tools_pt, and unit helper.
Made-with: Cursor
