* fix(guardrails): skip embeddings input in google_text_moderation and openai_moderations
Since v1.103.0 the proxy dispatches these legacy moderation callbacks on every route, and they read the embeddings input field as text, so benign embedding inputs were blocked. Return early for embedding call types, matching the Aim and Cato guardrails.
* test(guardrails): type the embeddings moderation test parameters
* fix(guardrails): keep openai_moderations moderating embeddings, skip only google_text_moderation
Google's moderation scores topic categories like Finance and Health, so it blocks
documents being indexed by subject. OpenAI's moderation scores harm categories only,
so it keeps moderating embedding input as it has since v1.103.0
* fix(guardrails): skip Gigachat passthrough embeddings in google_text_moderation
Treat an allm_passthrough_route call whose endpoint ends in embeddings as
non-conversational, log the skip, return None from the early exit, and type
the hook's data so the basedpyright gate does not grow
* fix(guardrails): match only plain embeddings passthrough paths in google_text_moderation
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)
* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)
* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts
* fix: restore pass-through behavior on paths the typing pass tightened
MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites
* fix(runwayml): keep a malformed 2xx task body on the 500 path
* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report
EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.
`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.
The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.
* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed
* test: annotate the new regression tests' locals with Final
* test: annotate the remaining new test locals with Final
Response cache reads and writes open cache.get llm_response and cache.set llm_response phase spans with their Redis spans nested underneath, on the Python path and on the native Rust path, and deployment selection runs inside a route {model_group} phase so the cooldown, usage and model-id reads the router issues nest under it before chat {model}. The autorouter classifier call nests under that route phase as well and carries its typed internal origin on litellm.request.purpose, so it is told apart from the provider attempt. Service spans are named {service}.{verb} {target} from a low-cardinality key family the producer declares (llm_response, auth_objects, spend_counters, router_cooldowns, claude_code_session_router_binding, rate_limits, pod_lock, budget_reset, ...) instead of the raw method or a per-request pipeline length; a pipeline flush is targeted by the one family its ops share or by mixed with the sorted families on litellm.redis.families, a batch op keeps the family it was declared under whichever pipeline or standalone read settles it, and the ambient family labels Redis spans only, never the DB write-back a task spawned inside that context performs later. The raw method stays on litellm.service.call_type and on the Prometheus and Datadog labels. Caller attribution is carried across asyncio task boundaries on a ContextVar so forwarder-only chains no longer surface, the raw cache key is dropped from Redis span metadata, pipeline op counts land as an integer attribute, every call_type the Redis cache layer emits maps to a verb, and a scan over litellm/ and enterprise/ fails when a Redis producer, batch reservation included, declares no key family.
A V2 logger built for a key or team logging entry while the operator's V2 logger is already registered keeps only the exporters its own preset contributed, whether or not the operator holds credentials for that backend, so every chat span no longer reaches the operator's collector twice. A span the success callback has to open itself, with no pre-call carrier, starts at the provider handoff (api_call_start_time) instead of the logging object's creation.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): scan batch records with the content hooks that are not guardrails
Guardrails were made to run on batch uploads by scanning each record through the pre-call hook
with the walk limited to guardrails. That limit exists because the same branch carries the rate
limiters and budget accounting, which must count an upload once rather than once per line. It
also excluded every enforcement hook written as a plain CustomLogger, so prompt-injection
detection, Azure content safety, banned keywords and the blocked-user check never saw a batch
record at all. Content that is a hard 400 online reached the provider verbatim through batch.
A CustomLogger now declares whether its pre-call hook judges the payload or merely counts the
request. The four that judge it opt in, the walk admits them, and both short-circuits learn
about them, including the one that decides whether the file is streamed off disk in the first
place: a proxy configured only with one of these hooks was skipping the scan entirely. Nothing
that counts a request is marked, so an upload still costs one slot and one budget check.
* refactor(proxy): drop the per-hook comment the attribute contract already states
* test(proxy): make the classification a ledger, and pin the wiring with a real hook
The classification test listed the two non-enterprise hooks by hand, so unmarking either
enterprise one changed nothing and the mutation matrix passed with both surviving. It now walks
the hook registries and fails on any pre-call CustomLogger that is on neither side, which also
gives the flag the forcing function it lacked: an enforcement hook added later would otherwise
default to off and silently skip batch records, which is the bug being fixed here.
Nothing exercised the path the bug actually lived on either, since every test raised its own
exception rather than a real hook's. One test now drives the shipped prompt-injection hook
through the scan, which pins the part no synthetic exception reaches: a chained exception reads
as a failure to judge, so refactoring any of these hooks to `raise ... from` would turn every
per-record drop into an aborted upload.
Also records why a hook that rewrites the payload for routing stays unmarked, and that only the
leaf class is consulted.
* test(proxy): set the callback list through monkeypatch rather than writing the global
The hooks gated on ``call_type == "completion"`` but the proxy ingress
passes ``route_type`` straight through as ``call_type`` —
``"acompletion"`` for /v1/chat/completions and ``"aresponses"`` for
/v1/responses. Tests passed because they used the literal sync
``"completion"`` value, masking the gap.
Switch both hooks to ``is_text_content_call_type`` (matches the
canonical runtime values: completion / acompletion / aresponses) and
update existing tests to assert against runtime values, plus parametrize
a regression test that pins the gate.
Greptile P2 follow-ups on _content_utils.py:
- Drop unreachable ``_resolve_messages``. The new
``_iter_inspection_messages`` walks ``messages`` AND ``input``
independently; leaving the old fallback-only variant around invited a
future maintainer to wire it back up and silently narrow coverage.
- Rename ``iter_user_text`` → ``iter_message_text``. The helper walks
every role (user, assistant, system); the old name implied user-turn
content only. Callers and tests updated.
- Close mixed-list coverage gap. When ``data["input"]`` was a list
mixing content-part dicts and bare strings, ``iter_message_text`` and
``build_inspection_messages`` only saw the dict parts while
``walk_user_text`` already inspected both. ``_iter_text_parts_in_content``
now treats bare strings inside a content list as text fragments, so
read and write helpers agree on coverage.
Adds two regression tests for the mixed-list shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several guardrail hooks short-circuit when ``message.content`` is a list
or when the request uses the Responses-API ``input`` field instead of
``messages``. Centralise the content-walking logic in a shared helper and
update the affected hooks so list-format and Responses-API payloads no
longer skip inspection.
Also: Aim's ``async_post_call_success_hook`` now inspects every choice
(via ``asyncio.gather``) instead of only ``choices[0]`` — the prior
behaviour let ``n>1`` callers hide content in subsequent completions.
Hooks updated to use the new helper:
- aim, lakera_ai_v2, lasso (post a synthesised messages list to a remote
guardrail service)
- azure_content_safety, ibm_detector, banned_keywords, openai_moderation,
google_text_moderation (iterate text fragments locally)
- secret_detection (walk-and-rewrite to redact in place)
Drive-by fix: the legacy ``data["prompt"]`` list-handling path in
secret_detection rebound the loop variable instead of mutating the list,
leaving secrets unredacted on text-completion calls; corrected to index
back into the list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(test_router.py): initial unit test confirming router.afile_content uses dynamic api key / api base
* fix(managed_files.py): filter deployments for only those within file id mapping
ensure call works - only route to models where the file was written
* fix(proxy_server.py): fix loading in model ids from config, if config id is int
* fix(router.py): return all model file id mappings on create_file
if multiple deployments - this ensures all the file id mappings are bubbled up
Fixes issue when trying to use loadbalanced deployments - only 1 file id mapping was being stored
* fix: cleanup print statement
* feat(managed_files.py): add auth check on managed files
Implemented for file retrieve + delete calls
* feat(files_endpoints.py): support returning files by model name
enables managed file support
* feat(managed_files/): filter list of files by the ones created by user
prevents user from seeing another file
* test: update test
* fix(files_endpoints.py): list_files - always default to provider based routing
* build: add new table to prisma schema
* Add LiteLLM Managed file support for `retrieve`, `list` and `cancel` finetuning jobs (#11033)
* feat: initial commit adding managed file support to fine tuning endpoints
* feat(fine_tuning/endpoints.py): working call to openai finetuning route
Uses litellm managed files for finetuning api support
* feat(fine-tuning/main.py): refactor to use LiteLLMFineTuningJob pydantic object
includes 'hidden_params'
* fix: initial commit adding unified finetuning id support
return a unified finetuning id we can use to understand which deployment to route the ft request to
* test: fix test
* feat(managed_files.py): return unified finetuning job id on create finetuning job
enables retrieve, delete to work with litellm managed files
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(fine_tuning_endpoints/endpoints.py): add managed files support to list finetuning jobs
* feat(finetuning_endpoints/main): add managed files support for retrieving ft job
Makes it easier to control permissions for ft endpoint
* LiteLLM Managed Files - Enforce validation check if user can access finetuning job (#11034)
* feat: initial commit adding managed file support to fine tuning endpoints
* feat(fine_tuning/endpoints.py): working call to openai finetuning route
Uses litellm managed files for finetuning api support
* feat(fine-tuning/main.py): refactor to use LiteLLMFineTuningJob pydantic object
includes 'hidden_params'
* fix: initial commit adding unified finetuning id support
return a unified finetuning id we can use to understand which deployment to route the ft request to
* test: fix test
* feat(managed_files.py): return unified finetuning job id on create finetuning job
enables retrieve, delete to work with litellm managed files
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(fine_tuning_endpoints/endpoints.py): add managed files support to list finetuning jobs
* feat(finetuning_endpoints/main): add managed files support for retrieving ft job
Makes it easier to control permissions for ft endpoint
* feat(managed_files.py): store create fine-tune / batch response object in db
storing this allows us to filter files returned on list based on what user created
* feat(managed_files.py): Ensures users can't retrieve / modify each others jobs
* fix: fix check
* fix: fix ruff check errors
* test: update to handle testing
* fix: suppress linting warning - openai 'seed' is none on azure
* test: update tests
* test: update test
* feat: initial commit adding managed file support to fine tuning endpoints
* feat(fine_tuning/endpoints.py): working call to openai finetuning route
Uses litellm managed files for finetuning api support
* feat(fine-tuning/main.py): refactor to use LiteLLMFineTuningJob pydantic object
includes 'hidden_params'
* fix: initial commit adding unified finetuning id support
return a unified finetuning id we can use to understand which deployment to route the ft request to
* test: fix test
* feat(managed_files.py): return unified finetuning job id on create finetuning job
enables retrieve, delete to work with litellm managed files
* test: update test
* fix: fix linting error
* fix: fix ruff linting error
* test: fix check
* feat(model_info_view.tsx): enable updating model info for existing models on UI
Fixes LIT-154
* fix(model_info_view.tsx): instantly show model info updates on UI
* feat(proxy_server.py): enable flag on `/models` to include model access groups
This enables admin to assign model access groups to keys/teams on UI
* feat(ui/): add model access groups on ui dropdown when creating teams + keys
* refactor(parallel_request_limiter_v2.py): Migrate multi instance rate limiting to OSS
Closes https://github.com/BerriAI/litellm/issues/10052
* fix(router.py): write file to all deployments
allows unified file id to work across multiple deployments
* fix(view_logs/index.tsx): show call type in request logs
* fix(router.py): pass a deep copy of kwargs to avoid conflict across multiple runs
* fix(batch_utils.py): broaden check
* fix(router_utils.py): handle null type for function name
* fix(proxy_track_cost_callback.py): fix ruff check error
* fix(router.py): handle healthy_deployments as a dict
* feat(managed_files.py): support encoding / decoding unified batch id … (#10711)
* feat(managed_files.py): support encoding / decoding unified batch id when using managed files
allows routing retrieve batch to the right model id
* fix: fix linting error
* feat(managed_files.py): support unified output file id
enables batch output file id to be used to retrieve the actual file
* fix(managed_files.py): attempt to fix ci/cd linting error
* fix: fix ruff check
* fix(router.py): write file to all deployments
allows unified file id to work across multiple deployments
* fix(view_logs/index.tsx): show call type in request logs
* fix(router.py): pass a deep copy of kwargs to avoid conflict across multiple runs
* fix(batch_utils.py): broaden check
* fix(router_utils.py): handle null type for function name
* fix(proxy_track_cost_callback.py): fix ruff check error
* fix(router.py): handle healthy_deployments as a dict
* feat(managed_files.py): support encoding / decoding unified batch id … (#10711)
* feat(managed_files.py): support encoding / decoding unified batch id when using managed files
allows routing retrieve batch to the right model id
* fix: fix linting error
* test: add unit tests
* fix: fix ruff check
* refactor(managed_files.py): move enterprise feature into enterprise folder
prevent unexpected surprises
* refactor: safely handle enterprise hooks
* fix: fix ruff check errors
* fix(files_endpoints.py): cleanup enterprise code from OSS
* refactor: complete cleanup
* fix(managed_files.py): complete cleanup
* fix(managed_files.py): instrument to be able to update deployment values post-router selection and just before making llm call
* fix(managed_files.py): instrument to be able to update deployment values post-router selection and just before making llm call
* fix: fix linting error
* fix: fix linting error
* fix: initial commit of v2 parallel request limiter hook
enables multi-instance rate limiting to work
* fix: subsequent commit with additional refactors
* fix(parallel_request_limiter_v2.py): cleanup initial call hook
simplify it
* fix(parallel_request_limiter_v2.py): working v2 parallel request limiter
* fix: more updates - still not passing testing
* fix(test_parallel_request_limiter_v2.py): update test + add conftest
* fix: fix ruff checks
* fix(parallel_request_limiter_v2.py): use pull via pattern method to load in keys instance wouldn't have seen yet
Fixes issue where redis syncing was not pulling key until instance had seen it
* test: update testing to cover tpm and rpm
* fix(parallel_request_limiter_v2.py): fix ruff errors
* fix(proxy/hooks/__init__.py): feature flag export
* fix(proxy/hooks/__init_.py): fix linting error
* ci(config.yml): add tests/enterprise to ci/cd
* fix: fix ruff check
* test: update testing
* add session id in spendLogs
* fix log proxy server request as independant field
* use trace id for SpendLogs
* add _ENTERPRISE_ResponsesSessionHandler
* use _ENTERPRISE_ResponsesSessionHandler
* working session_ids
* working session management
* working session_ids
* test_async_gcs_pub_sub_v1
* test_spend_logs_payload_e2e
* working session_ids
* test_get_standard_logging_payload_trace_id
* test_get_standard_logging_payload_trace_id
* test_gcs_pub_sub.py
* fix all linting errors
* test_spend_logs_payload_with_prompts_enabled
* _ENTERPRISE_ResponsesSessionHandler
* _ENTERPRISE_ResponsesSessionHandler
* expose session id on ui
* get spend logs by session
* add sessionSpendLogsCall
* add session handling
* session logs
* ui session details
* fix on rowExpandDetails
* ui working sessions
* build(pyproject.toml): add new dev dependencies - for type checking
* build: reformat files to fit black
* ci: reformat to fit black
* ci(test-litellm.yml): make tests run clear
* build(pyproject.toml): add ruff
* fix: fix ruff checks
* build(mypy/): fix mypy linting errors
* fix(hashicorp_secret_manager.py): fix passing cert for tls auth
* build(mypy/): resolve all mypy errors
* test: update test
* fix: fix black formatting
* build(pre-commit-config.yaml): use poetry run black
* fix(proxy_server.py): fix linting error
* fix: fix ruff safe representation error
* fix(proxy_server.py): use default azure credentials to support azure non-client secret kms
* fix(langsmith.py): raise error if credentials missing
* feat(langsmith.py): support error logging for langsmith + standard logging payload
Fixes https://github.com/BerriAI/litellm/issues/5738
* Fix hardcoding of schema in view check (#5749)
* fix - deal with case when check view exists returns None (#5740)
* Revert "fix - deal with case when check view exists returns None (#5740)" (#5741)
This reverts commit 535228159b.
* test(test_router_debug_logs.py): move to mock response
* Fix hardcoding of schema
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* fix(proxy_server.py): allow admin to disable ui via `DISABLE_ADMIN_UI` flag
* fix(router.py): fix default model name value
Fixes 55db19a1e4 (r1763712148)
* fix(utils.py): fix unbound variable error
* feat(rerank/main.py): add azure ai rerank endpoints
Closes https://github.com/BerriAI/litellm/issues/5667
* feat(secret_detection.py): Allow configuring secret detection params
Allows admin to control what plugins to run for secret detection. Prevents overzealous secret detection.
* docs(secret_detection.md): add secret detection guardrail docs
* fix: fix linting errors
* fix - deal with case when check view exists returns None (#5740)
* Revert "fix - deal with case when check view exists returns None (#5740)" (#5741)
This reverts commit 535228159b.
* Litellm fix router testing (#5748)
* test: fix testing - azure changed content policy error logic
* test: fix tests to use mock responses
* test(test_image_generation.py): handle api instability
* test(test_image_generation.py): handle azure api instability
* fix(utils.py): fix unbounded variable error
* fix(utils.py): fix unbounded variable error
* test: refactor test to use mock response
* test: mark flaky azure tests
* Bump next from 14.1.1 to 14.2.10 in /ui/litellm-dashboard (#5753)
Bumps [next](https://github.com/vercel/next.js) from 14.1.1 to 14.2.10.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.1.1...v14.2.10)
---
updated-dependencies:
- dependency-name: next
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* [Fix] o1-mini causes pydantic warnings on `reasoning_tokens` (#5754)
* add requester_metadata in standard logging payload
* log requester_metadata in metadata
* use StandardLoggingPayload for logging
* docs StandardLoggingPayload
* fix import
* include standard logging object in failure
* add test for requester metadata
* handle completion_tokens_details
* add test for completion_tokens_details
* [Feat-Proxy-DataDog] Log Redis, Postgres Failure events on DataDog (#5750)
* dd - start tracking redis status on dd
* add async_service_succes_hook / failure hook in custom logger
* add async_service_failure_hook
* log service failures on dd
* fix import error
* add test for redis errors / warning
* [Fix] Router/ Proxy - Tag Based routing, raise correct error when no deployments found and tag filtering is on (#5745)
* fix tag routing - raise correct error when no model with tag based routing
* fix error string from tag based routing
* test router tag based routing
* raise 401 error when no tags avialable for deploymen
* linting fix
* [Feat] Log Request metadata on gcs bucket logging (#5743)
* add requester_metadata in standard logging payload
* log requester_metadata in metadata
* use StandardLoggingPayload for logging
* docs StandardLoggingPayload
* fix import
* include standard logging object in failure
* add test for requester metadata
* fix(litellm_logging.py): fix logging message
* fix(rerank_api/main.py): fix linting errors
* fix(custom_guardrails.py): maintain backwards compatibility for older guardrails
* fix(rerank_api/main.py): fix cost tracking for rerank endpoints
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: steffen-sbt <148480574+steffen-sbt@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
