* fix(bedrock): accept Converse messages with no content key
A user or tool message whose content key is missing (or null, which the
message cleanup strips) made every Bedrock Converse request fail with
APIConnectionError 'content' before reaching Bedrock. The Converse
transform now reads content with .get for those messages, as it already
did for assistant messages: a content-less user message adds no block
and a content-less tool message becomes a toolResult with empty content.
The str branch also sends the continue message text instead of the
original whitespace-only text.
* fix(bedrock): send the continue message for a content-less user turn
* test(bedrock): type the content-less Converse message test parameters
* fix(bedrock): accept a Converse system message with no content key
* refactor(bedrock): read the system message content with get
* test(integration): audit Bedrock Converse messages without content
Adds the /audit cells for a chat message whose content key is missing or
null on a Converse-routed Bedrock deployment: happy, sad, edge, and chaos
rows through the OpenAI SDK, the Anthropic SDK, and raw httpx against the
scripted upstream, asserting the caller's response, the body the peer
received, and the spend row. The owned-proxy readiness deadline in the
integration harness is now INTEGRATION_PROXY_READY_SECONDS (default 70).
* test(integration): bound stray spend rows in the mid-burst restart cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(s3_v2): add s3_partition_granularity option for hourly S3 folders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover s3 v2 partition granularity across surfaces, settings and chaos
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover previous_response_id history rebuilt from an hourly cold storage object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): reuse the cold storage key only when s3_v2 owns cold storage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): cover hour rollover, postgres outage, in-flight switches, key/team vars and real S3 layout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(liccheck): authorize libfaketime, the GPLv2 dev-only clock the s3 rollover integration test preloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): wait for the rejected-request cell's payloads by id, not by line count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): declare the postgres outage cell's models in config and trip the relay on burst ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): drop the libfaketime hour rollover cell and its dev dependency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): deselect the s3_v2 live e2e on the stage-mirror stack
The stage-mirror config enables no s3_v2 callback, so every test in test_s3_log_e2e.py fails its readiness check there. The file keeps running in the Buildkite e2e lane, which configures s3_v2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): declare the sink outage burst models in config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(s3_v2): read cold storage metadata without an empty dict default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for the proxy to reconnect before the postgres outage recovery request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* test(ci): add used_client_oauth_token to the GCS pub/sub spend-log golden
#43063 stamps used_client_oauth_token into spend-log metadata, so
test_async_gcs_pub_sub_v1 failed on main with an extra metadata key
* test(ui): give the auto-router threshold save wait room for the availability debounce
#42625 keeps Save disabled while a 300ms-debounced availability check runs.
This test waits for Save right after the change, so the whole debounce lands
inside waitFor's 1s default and it times out under CI load. It is the
recurring UI Unit Tests failure on main since #42625 landed
* test(e2e): expect no pricing tier on bills for streamed calls OpenAI served at default
#42870 added both the rule that a served default or standard tier bills at
base pricing and records no service_tier, and streamed tests expecting the
row to record 'default'. They have failed on every scheduled litellm-e2e run
since. The tests now map the served tier to the pricing basis the bill must
record and check input is billed at that basis's rate; the messages case
registers custom rates so the rate check has something to compare against
* test(e2e-ui): wait for the call-id search before hovering the logs row
The row the spec hovers is already on the unfiltered first page, so it was
found before the search request returned. The search response then
re-rendered the table under the mouse, and the Base UI tooltip never opened.
Reproduced with Playwright against a local proxy: hovering right after the
fill never shows the tooltip, hovering after the search response shows the
call id every time
* test(e2e): run the Together structured-output case on the hybrid Qwen with reasoning off
The case picked the cheapest Together row flagged supports_response_schema.
DeepSeek-V4-Flash-0731 hit its cost-map deprecation date on 2026-09-29, so the
pick moved to GLM-5.3-Flash, a reasoning-only model that spends the 1024-token
budget thinking and returns content=None. Qwen3.5-9B is the pinned hybrid model
the reasoning_effort=none case already exercises, and Together lists it with
structured output support
* test(integration): read the agent 365 guardrail status by its own name in spend logs
The MCP shard runs under xdist against one database, and a sibling file creates a
default_on pre_mcp_call content filter there. The owned proxy reloads DB guardrails, so
that filter's 'success' entry could land first in guardrail_information and the test
read it instead of the agent 365 verdict
* test(unit): ignore asyncio's leaked-task records in the budget limiter push-failure log check
gc.collect() inside the caplog window can collect a pending task an earlier test left
on a closed loop, and asyncio logs 'Task was destroyed but it is pending' into this
test's records. The check still counts every LiteLLM logger, and unretrieved task
exceptions on this loop still go through the asserted exception handler
* test(e2e-ui): fill the create-tag fields inside the dialog
#42949 added 'Filter by tag name' and 'Filter by description' inputs to the Tag
Management page, so page-wide getByLabel('Tag Name') and getByLabel('Description')
match two elements and Playwright's strict mode fails the create step
* test(integration): run integration proxies with the CI license
Multi-worker proxies start each uvicorn worker in a fresh process, so every
worker reads the license from its environment. Forward LITELLM_LICENSE into the
proxy and test runner environments
* ci: save GitHub Actions caches only from main and bump codecov-action to 5.5.5
Every pull request saved its own uv, maturin, Rust and Prisma caches, about
4.5 GB per PR, so the repository's 10 GB cache budget evicted main's entries
within minutes. Pull request jobs then missed every cache, downloaded all
dependencies from PyPI and hit the install step timeouts. Pull requests now
restore only, and main keeps the caches warm for them. test-linting and
check-ui-api-types run only on pull requests and keep saving
codecov-action 5.5.4 imports its signing key from the deleted codecovsecurity
keybase account, so every upload failed signature verification. 5.5.5 reads it
from codecovsecops; the key ID matches the one signing the current CLI
* test(unit): join the session-minting thread before collecting the handler
asyncio.to_thread resumes the test as soon as the worker sets its result,
while the pool thread can still hold the work item and through it the
handler. gc.collect() then cannot finalize the handler and the session stays
open. A pool that shuts down before the test continues drops that reference
* test(integration): relaunch owned proxies that lose their port, expire idle gateway connections early
owned_proxy_process released its reserved port and the proxy bound it only
after full startup, so another xdist worker or an outgoing connection could
take it first and the proxy exited with 'address already in use'. The launch
now retries on a fresh port when that happens and stops every failed attempt.
uvicorn closes idle keep-alive connections after 5 seconds and httpx expired
them at the same 5 seconds, so a request sent right at that mark could reuse a
socket the server was closing and get 'Connection reset by peer'. Gateway
clients now drop idle connections after 2 seconds
* ci(circleci): give the base SDK wheel build the same 30 minute no-output window as the Windows build
The release profile builds with fat LTO and one codegen unit, so the final
link of litellm-cache-s3 runs silently for minutes. Successful builds take
711 to 749 seconds, right at the default 10 minute no-output limit, and about
30% of recent runs were killed there
* test(integration): model the budget-reset database outage as 10 seconds instead of 5 refused connections
The proxy retries the database about every 30 seconds and each retry opens
roughly one connection, so a 5-connection outage took 3 to 4 retries to clear
and recovery landed between 60 and 90 seconds, straddling the test's 80 second
reset window. A fixed 10 second outage still refuses the immediate reconnect
and recovers on the next retry
* ci: move the unit-test uv cache split into a composite action
check_workflow_startup_safety sums every setup step's timeout, so the save and
restore variants each counted 5 minutes although only one runs. One composite
step keeps the setup ceiling at 35 minutes
* test(unit): point tiktoken at the bundled cache for every unit test
The rust_bridge tokenizer tests loaded o200k_base before any test in their
xdist worker had imported default_encoding, so tiktoken fell back to the
temp cache and tried to download under pytest-socket. Move the session
fixture from litellm_core_utils/conftest.py to the root unit conftest.
* test(integration): answer model discovery probes in the hosted_vllm wire tests
The router's periodic upstream model info refresh sends GET /v1/models to
hosted_vllm deployments, so a wire server that is live during a refresh
sees an extra request. Answer the probe with an empty model list and leave
it out of the provider-call assertions, matching the responses bridge
tests.
* feat(proxy): record in spend logs whether a request used a client-forwarded Anthropic OAuth token
Stamp metadata.used_client_oauth_token where the proxy decides to forward a
client's Anthropic OAuth token, carry it through StandardLoggingMetadata into
the spend log row, add a used_client_oauth_token filter to /spend/logs/ui, and
surface it on the Logs page as a Credential filter and drawer field. The token
itself never reaches the log
* fix(proxy): carry used_client_oauth_token onto failure spend rows for litellm_metadata routes
* fix(proxy): resolve used_client_oauth_token against the provider the call was sent to
* fix(proxy): keep the proxy's used_client_oauth_token stamp on failure rows and move the resolver under llms/anthropic
* fix(logging): read used_client_oauth_token from the proxy-stamped metadata slot
On routes that carry proxy metadata in litellm_metadata, metadata is the
caller's own body field, and merge_litellm_metadata lets it win. Resolve the
flag from litellm_metadata when the proxy stamped it there so a caller cannot
set it in the standard logging payload
* fix(spend-logs): read used_client_oauth_token from the bucket the route stamped
A guardrail on the unified path adds litellm_metadata to a chat request after
the proxy stamped metadata, so both spend row writers read the new bucket and
stored null. The success row now resolves the flag the same way the callback
payload does, and the failure row picks the bucket from the request route.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): inherit catalog service-tier rates for custom-priced deployments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): apply tier-suffixed long-context rates when only tier thresholds are set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover canonical cost-map backend model resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): use descriptive names for service-tier pricing fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(transcription): honor base_url alias for Groq Whisper and report it as the api base
transcription() and speech() only accepted api_base, so a deployment configured with base_url leaked the alias into the provider params, which Groq rejected as an unknown param, and the request never reached the internal gateway. get_api_base() now reads the same alias so response headers and logs show the configured endpoint instead of the provider default
Resolves LIT-9071
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(speech): keep base_url after existing audio params, route Vertex speech to it, skip empty alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): keep the provider's served service_tier on streamed chunks and spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): satisfy type-discipline and strict ruff budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): stamp the served service_tier on every Responses bridge chunk
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-adapter): expose streamed chunks so disconnects bill partial spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(service-tier): cover anthropic and responses served-tier billing paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-adapter): return a chunks-exposing stream so disconnects bill partial spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(service-tier): bill disconnects through the router's anthropic stream wrapper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: apply ruff format to the anthropic stream changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coverage): ignore delegating properties the ast scan cannot see
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: keep the cast-ok reasons on the cast call line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover served service_tier billing for streamed chat and messages, complete and disconnected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-cache): delegate chunks/messages/model through the messages stream cache writer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): keep service_tier on OpenAI-compatible parsed chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(streaming): parameterize delegated chunks and messages types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): follow the anthropic pass_through rename after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drain the logging worker between response cache tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): cover azure, databricks, responses bridge and gemini served tiers in the stream billing integration test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): keep the served service_tier on streamed chunks and bill it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): type the served service_tier chunk without a loose kwargs dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): bill the served service_tier over the requested one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): drop explanatory comment from the tier resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(model-prices): correct azure/eu/gpt-6-astra to Data Zone rates
Co-authored-by: rain <1504569896@qq.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model-prices): align groq, gemini and openai entries with official docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model-prices): roll in verified Vertex, Gemini, OpenRouter and Azure AI registry fixes
Absorbs the fields from #43609, #43666, #43671 and #43644 that match the provider's own docs or price API today, and adds a cost test for the azure/eu/gpt-6-astra Data Zone tiers
Co-authored-by: bunnysayzz <stfuazzo@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model-prices): add Copilot, Bedrock Kimi K3, Gemini Robotics and OpenRouter values from official sources
Co-authored-by: Michal Formanek <michal.formanek@generaliceska.cz>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: rain <1504569896@qq.com>
Co-authored-by: bunnysayzz <stfuazzo@gmail.com>
Co-authored-by: Michal Formanek <michal.formanek@generaliceska.cz>
* feat(cost_calculator): add cost_per_second for chat per-second pricing
Keep legacy input_cost_per_second and output_cost_per_second as aliases for chat, completion, embedding and responses. When both legacy fields are set, input_cost_per_second wins
Move Bedrock commitment rows to cost_per_second so they bill once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost_calculator): drop legacy per-second fields from chat paths
Keep Azure chat token pricing generic and update inert Voxtral rates and SageMaker examples
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): recognize output-only per-second rates
Include output_cost_per_second when checking whether a deployment cost entry has pricing so output-only legacy aliases remain attached to the deployment during cost selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(pricing): cover cost_per_second and legacy per-second aliases through the proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost_calculator): drop output_cost_per_second as a chat per-second alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost_calculator): restore output_cost_per_second as a chat per-second fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): keep input_cost_per_second on bedrock commitment rows for older clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): emit the reasoning item on streaming /v1/responses for signature-only thinking
Anthropic models return thinking blocks with empty text and the reasoning carried in the
signature: Claude Fable 5.1 and Claude Opus 5.5 by default, and Bedrock adaptive thinking
with or without an effort. On streaming /v1/responses the chat->Responses bridge opened a
reasoning output item only on reasoning_content text
(LiteLLMCompletionStreamingIterator._ensure_output_item_for_chunk), and
ChunkProcessor.get_combined_thinking_content kept an assembled thinking block only when it
had thinking text. Such a response emitted no reasoning item mid-stream and none in
response.completed, so a streaming Responses client could not replay the reasoning even
though the reasoning tokens were billed. Non-streaming /v1/responses was unaffected.
Open the reasoning item when the delta carries a signed or redacted thinking block, and
keep a signed block through stream assembly even when its thinking text is empty.
Unsigned text-only fragments are still dropped. The reasoning-text path is unchanged.
(cherry picked from commit bc9b6f8a5c)
* test(vertex_ai): move orphaned gemma streaming tests into the llm-vertex-ai shard
PR #43147 left a copy of the Gemma streaming tests under
tests/test_litellm/llms, a tree no CI shard claims, which broke
assert-ci-coverage and assert-shard-coverage on main. Fold the two
streaming tests into the existing tests/unit/llms/vertex_ai file so the
llm-vertex-ai shard runs them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chloe Lu <chloe.lxd@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tools): salvage concatenated JSON tool call arguments
* fix(tools): harden concatenated tool-call salvage for review findings
Skip non-dict JSON during split so salvage cannot emit empty tool calls.
Collapse srvtoolu_ expansions to the first object so server results stay paired.
Allocate __concat_n ids that cannot collide with sibling tool call ids.
Propagate cache_control onto every expanded Anthropic tool_use block.
Rename the XML invoke loop variable so the key-leak gate no longer flags {args}
* test(tools): cover concat id bump and srvtoolu array keep
Only collapse srvtoolu_ when concatenated salvage expanded; a valid JSON
array argument stays one server tool input
* revert(anthropic): drop concat expansion from pass-through adapter
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
* revert(tools): keep concat salvage out of request-side tool converters
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
* fix(tools): expand strictly salvaged concatenated tool arguments in normalized tool calls
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
* fix(tools): retain at most the salvage cap while validating concatenated arguments
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
* test(tools): assert concat sibling ids unique after sanitization
A sibling id that only collides after colon-to-underscore sanitization must force the next concat suffix
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
* refactor(tools): drop unused strict mode from split_concatenated_json_objects
Strict mode had no production caller. Rejection cases now sit on salvage, and split matches upstream main
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
---------
Co-authored-by: Techboy bebop <kumarpriyanshu09@users.noreply.github.com>
_is_unsignable_thinking_block() only checked block["signature"], so a
thinking block with a valid-looking signature but empty (or
whitespace-only) thinking text sailed through _drop_unsignable_thinking_blocks
and into anthropic_messages_pt(). Anthropic rejects that with:
400 messages.N.content.M.thinking: each thinking block must contain thinking
This is reachable whenever a thinking_blocks history item gets replayed
through this Anthropic-shaped request path (e.g. a non-Anthropic reasoning
turn with no summary text), the same class of bug PR #36033 fixed on the
Responses adapter's own separate code path.
Now the signature check runs first (unsigned blocks are still dropped, same
as before), then an additional check drops the block if `thinking` is
missing, not a string, or strips to empty. redacted_thinking blocks are
untouched since they don't have type == "thinking".
* feat(otel): add SigNoz preset for OpenTelemetry v2
Adds the signoz callback (OTLP/HTTP exporter, GenAI vocabulary, key and team level dynamic ingestion endpoint and key) as an OpenTelemetry v2 preset, with the preset factory accepting the allow_missing_credentials kwarg the V2 registry always passes so construction no longer falls back silently to legacy OpenTelemetry. Ships the deterministic tests/integration/observability/test_signoz_delivery.py audit suite
Absorbs the work from https://github.com/BerriAI/litellm/pull/38206
Co-authored-by: Nagesh Bansal <nageshbansal59@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop explanatory comments from the SigNoz preset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(types): keep signoz dynamic param lines within ruff format width
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(signoz): assert the missing-endpoint boot path directly instead of in an except block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for the signoz health service
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): allowlist SigNoz key/team endpoints and route keyless collectors without the operator key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): terminate the SigNoz shutdown cell before the flush and drop test docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): keep the shared tenant routing untouched and require an ingestion key for SigNoz key/team endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): warn about a keyless SigNoz team endpoint from the header resolver so the shared cache actually reaches it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Nagesh Bansal <nageshbansal59@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langtrace): integration test for the built-in callback wire (path, x-api-key)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langtrace): deliver spans to app.langtrace.ai/api/trace with x-api-key
The built-in langtrace callback posted to the dead host langtrace.ai, sent
the key as api_key instead of x-api-key, and let the OTLP endpoint
normalizer append /v1/traces to the complete /api/trace path, so every
export returned 404. Default the host to https://app.langtrace.ai, honor
LANGTRACE_API_HOST for self-hosted servers, pass the key as an exporter
header instead of a process-wide env var, and keep the /api/trace path
unchanged for traces on the langtrace callback only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langtrace): keep LANGTRACE_API_HOST that already ends in /api/trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langtrace): deterministic audit inventory for the built-in callback (surfaces, failures, endpoints, chaos)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langtrace): build the repeated-request body once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langtrace): outage cell asserts at-most-once delivery, not span loss
The OTLP HTTP exporter reposts once on ConnectionError and the batch
processor may still be flushing the previous burst when the sink closes,
so whether the outage burst is lost or delivered after revival depends
on timing. The invariant is no duplicate and recovery on the same port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langtrace): delivered spans must carry the prompt in the gen_ai.content.prompt event
The upstream echoes the marker into the completion, so a whole-span
match alone would still pass if the prompt event disappeared
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langtrace): disable model info refresh so the scripted upstream only sees completion requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(params): validate stream_chunk_size once and carry it as typed control options
Checks stream_chunk_size at the top of completion() and acompletion(), accepts
digit strings, returns a 400 naming the param unless drop_params is set, and
stores the checked value under _litellm_control. Bedrock Converse and Invoke
read it from litellm_params; the Bedrock-only checker and the dead Invoke pops
are gone. Owned-kwarg filtering now runs through one helper everywhere.
Refs LIT-8317
* test(bedrock): drop tests for the removed stream_chunk_size_from helper
Refs LIT-8317
* fix(params): check stream_chunk_size before the MCP gateway branch
Refs LIT-8317
* fix(params): return assert_never in the exhaustive control-options match
Refs LIT-8317
* fix(params): address council review of the control options change
Read all_litellm_params live so names registered after import stay
LiteLLM-owned, make litellm_params a required keyword on the stream
wrapper hooks, give digit strings and ints the same 18-digit range,
share the default-chunking test table, test the Responses bridge through
litellm.responses, and revert formatting-only churn in existing tests.
Refs LIT-8317
* fix(params): address the second council review of control options
Keep the Responses bridge on its original all_litellm_params forwarding,
narrow _int_from_decimal_string inline so it type-checks, bound nested
huge ints in the error message, store _litellm_control only when a value
is set, simplify the parser to its single field, drop the one-caller
wrapper, and tighten the tests.
Refs LIT-8317
* fix(params): keep the 18-digit length check on stream_chunk_size strings
A 19-character string with leading zeros such as 0000000000000000001 would
otherwise pass as 1, although the rule and the error message say at most
18 digits.
Refs LIT-8317
* test(params): tidy control options tests after council sign-off
Move the Responses bridge test into the existing bridge test file, drop the
rebind test that pinned an implementation detail, assert through
stored_control_options instead of the storage key, and cover
drop_params="true" through Bedrock streaming.
Refs LIT-8317
* test(params): wrap a chunking test row that went past 120 characters
Refs LIT-8317
* feat(guardrails): scan retrieved vector store chunks with the request's pre-call guardrails
Vector store retrieval runs inside acompletion after the proxy's pre-call
guardrails have already seen the request, so a retrieved chunk carrying an
injection reached the prompt unscanned. Each retrieved context message now
goes through every pre-call guardrail the request is subject to before it
is injected: a block raises the same 400 the guardrail gives for user text,
a masking guardrail rewrites the context, and a guardrail that fails while
scanning fails the request instead of injecting the chunk unscanned
* fix(guardrails): return a guardrail block unmapped from exception_type so the Responses API surfaces the guardrail's own 400
* fix(guardrails): build the deployment hooks' identity from stamped metadata only
Top-level user_api_key_* fields in a request body are client controlled, so the pre-call, chunk scan, and post-call deployment hooks now take UserAPIKeyAuth from the metadata the proxy stamped, and the chunk scanner returns or raises on every branch.
* fix(guardrails): block route verdicts on retrieved chunks, keep guardrail verdicts out of router retries and fallbacks, and scan chunks against the client's request
* fix(guardrails): keep the merged guardrail list when scanning chunks against the client's request
The scan request laid the client's kept body over the deployment kwargs, so a client
that sent its own top-level guardrails list shadowed the merged metadata.guardrails
list and a key or team guardrail skipped the chunk scan. The kwargs now win and the
keys the proxy relocates into metadata are dropped from the body's contribution.
* fix(guardrails): strip the deployment's guardrail keys from the chunk scan request so merged team guardrails still run
* test(guardrails): type the vector store scan test doubles
* test(guardrails): type the scan double's request data
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Register Sail (providers.json, LlmProviders.SAIL, OpenAI-compatible lists,
ProviderConfigManager) for chat, Responses and /v1/messages, and add its 12
models to both cost maps with asap, balanced and flex price columns.
Sail picks speed and price with metadata.completion_window and rejects
service_tier, so the Sail chat and Responses configs translate the tier:
default and priority to asap, flex to flex, balanced to balanced, auto to no
window. Billing prices the window that was sent. A tier Sail has no window
for, or a window or tier set where billing cannot see it (request metadata,
extra_body), is a 400 unless drop_params is set.
Add balanced to ServiceTier and its _balanced price columns to the model
info types, the Rust catalog and the dashboard schema. A transform_extra_body
hook on the chat and Responses base configs, which returns extra_body
unchanged by default, lets Sail keep the window when a caller also sends
extra_body.metadata. Sail is listed in the Add Model form and model picker.
Co-authored-by: shrey kharbanda <shrey@berri.ai>
* fix(otel): detach post-response service spans by request phase, name redis spans by operation
Service spans logged from the post-response phase (success callbacks, the response-cache write) now root their own trace linked to the request span even while the server span is still recording, instead of only when they happen to end after it. Redis service spans are named `redis <operation>`; the litellm call chain that issued them moves to the `litellm.service.caller` attribute via a typed `ServiceLoggerPayload.caller` field.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep the service caller on failure and legacy spans, test the production phase dispatch sites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): mark anthropic messages stream cache write as post-response phase
The /v1/messages streaming cache writer awaits async_add_cache inline
instead of going through create_cache_write_task, so its redis span
stayed parented under the request trace. Wrap the write in
post_response_phase so it detaches like the chat completions write.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): write the Messages stream cache in a background task after handoff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts
The Langfuse redaction unit test exports to a local OTLP capture instead of
polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill
test only requires spend rows for requests the surviving worker served. The
spend-routes sweep treats the intentional /spend/capture_rate 503 as expected.
CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper
instead of reading it as "variable exists". The integration egress guard now
matches the script's own cgroup, so it no longer blocks the CircleCI agent,
which runs as the same user.
* fix(ci): keep the policy-load backoff typed as float
* fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests
* fix(secrets): load CyberArk policy one request at a time per manager
* test(secrets): pin that non-conflict CyberArk policy failures are not retried
* test(unit): run tests/unit with only an allowlisted host environment
CircleCI's unit job inherits every project env var, so real provider keys,
REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that
assume none are set. Locally, litellm's import-time load_dotenv did the same
from any .env up the tree. The unit conftest now drops every variable outside
a small allowlist and disables dotenv before litellm is imported.
* test(e2e): name a failed search and the stuck batch status instead of misattributing them
The websearch session test read an empty web_search_tool_result_error block as a
successful search, so a failing search tool surfaced as a session billing bug.
The batch cancellation timeout now reports the last status the proxy returned.
* fix(ci): scrub the host environment per unit test instead of for the whole pytest process
GHA shards run tests/unit next to other suites in one process, so the import-time
scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP
upstream fell back to the SDK2 interpreter. The two websearch tests that called
OpenAI and Perplexity live are removed: tests/unit no longer sees their keys.
* fix(ci): scrub only the host variables present before litellm is imported
The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to
its bundled encodings, so tokenizer paths tried to download them and hit the
socket guard. The prisma setup test now passes its own database URL instead of
reading one another test leaked into the process environment.
* fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue
LoggingWorker marked a task done on whichever queue was current when the callback
finished, so a callback that outlived an event-loop change raised "task_done()
called too many times" or undercounted the new loop's queue. It now settles the
queue the task came from.
The rest are test isolation fixes for failures that only appeared when another
file ran first on the same xdist worker: a replaced user_api_key_cache, breaker
metrics unregistered by prometheus tests, semantic_router's health-check filter on
uvicorn.access, logging tasks carried over from bedrock tests, a Router-written
model_cost entry, and a stray post captured by the langflow test. The token
counter check now asserts bounded chunking instead of wall-clock time.
* test(e2e/ui): wait for the logout redirect before visiting a protected page
Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED).
* test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests
* test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read
* test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect
* test(integration/mcp): answer the model-info probe another test's proxy sends to the model double
* test: move key-gated tests/test_litellm SDK tests into tests/llm_translation and drop empty folders
* test: make token counter and health check unit tests run offline
* ci: point unit shards, rust path filter, Makefile and docs at tests/unit
* docs: fix stale test_litellm run paths in moved llm_translation tests
* fix: correct databricks e2e sys.path depth and contributing example path
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: load the embedding base image from a committed 100x100 PNG instead of downloading it
* test: move the volcengine embedding test into tests/unit
* test: check gpt2 and r50k_base tokenizer parity against committed tiktoken reference files
* test: check hub tokenizer selection against an in-memory Hugging Face hub
* test: serve image URLs from respx in the gemini tool-result and format-param tests
* ci: drop the emptied legacy core-utils test path
* test: cover the cohere and anthropic tokenizer paths in the hub tokenizer test
* test: fetch every format-param image through respx and check its bytes reach the request
* test: drop the gpt2 and r50k_base parity tests, which no litellm path uses
* test: drop comments that restate assertions in the format-param test
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rename fork-flag to unit-flag now that it applies on every event
* test: move tests/test_litellm root and small trees into tests/unit
Pure renames, no content changes. Follow-up commits in this PR fix
references, merge the three files that already existed in tests/unit,
keep live-provider tests in tests/test_litellm and wire CI.
* test: carry tests/test_litellm conftest isolation into tests/unit
Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS,
proxy-URL and keychain env, and session-end client cleanup now reset for
unit tests too. The environment isolation owns its MonkeyPatch so a test's
own monkeypatch is undone before the model-cost teardown runs.
* test: merge, split and prune the moved root and small-tree tests
Merge batches/test_batch_utils.py and the chat_completions and messages
dispatch tests into the files that already existed in tests/unit. Keep
the live Gemini interactions tests, the async image-fetch format test and
the OpenAI embedding scorer test in tests/test_litellm since they need
real network or keys. Put test_router.py under tests/unit/test_router so
the existing package no longer shadows it. Delete eight tests the audit
found superseded by stronger ones kept in this move.
* ci: run the moved root and small-tree tests under their legacy flags
Add the misc and responses-caching-types flags to unit_selection.sh and
CircleCI, extend enterprise-routing and mcp-integration, and point the
legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest
and change classifier at the new paths.
* test: make the new tests/unit directories packages
tests/unit/test_package_layout.py requires every directory to carry an
__init__.py, and without one the moved and retained
test_litellm_responses_bridge.py modules collide on import.
* test: scope the unit socket block to tests/unit in shared sessions
The GHA shards collect the legacy test-path and the unit selection in one
pytest session. The unit conftest's loopback-only block leaked into legacy
modules that reach the network at import. The legacy conftest now lifts the
restriction at collect and setup time, and the unit conftest re-applies it
when collecting its own modules.
* test: move tests/test_litellm/llms into tests/unit/llms
Rename-only. Moves the provider tests and the fine-tuning fixtures they
load, mirroring the old paths. Follow-up commits merge, split and wire them.
* test: merge, split and prune the moved llms tests
Merges the Databricks chat transformation tests into the existing unit
file, keeps the tests that need real keys or the network in
tests/test_litellm, deletes the audited tests a stronger unit test
already covers, and points imports at tests.unit.llms.
* ci: run the moved llms tests under their legacy flags
The Vertex AI and All Other Providers shards keep their legacy test-path
for the retained files and add the llm-vertex-ai and llm-other-providers
unit selections. CircleCI gets matching unit jobs.
* test: make the tests/unit/llms directories packages
Adds __init__.py to the moved dirs and drops the legacy ones whose
directories no longer hold tests.
* test: drop script runners and path hacks the llms split left dangling
The __main__ runners in the split openai_like files and the Databricks e2e
runner called tests that now live in the other half of the split or were
deleted. The retained legacy halves also no longer need sys.path edits.
* test: give the shard-script tests their own GITHUB_OUTPUT
They only passed where the runner set it. The CircleCI unit job's env
allowlist drops it, so the script's redirect failed there.
* test: point the router and module-deletion checks at tests/unit
router_code_coverage and code_qa_check_tests only searched tests/test_litellm,
so the moved router tests no longer counted. The two silent-experiment tests
the audit deleted were the only direct callers of those methods; they are
replaced with tests that assert the forwarded shadow request and the
recursion guard.
* test: move tests/test_litellm integrations and secret_managers into tests/unit
Rename-only. Mirrors the old paths, including the directory conftests
and the prompt and JSON fixtures. Follow-up commits prune and wire them.
* test: prune and repoint the moved integrations tests
Deletes the 7 audited tests a stronger test in the same tree already
covers, imports the TLS sink helpers from their new conftest path, and
restores os.environ after each integrations test. Some presets write
OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the
legacy tree's test ordering that header leaked into the AgentOps tests.
* ci: run the moved integrations tests under their legacy flag
The integrations GHA shard and a new CircleCI job run the integrations
unit selection. secret_managers joins the misc selection.
* docs: point integrations and secret_managers references at tests/unit
* test: make the moved integrations directories packages
* test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path
The Databricks e2e file is a manual script whose main() calls the tests
that were pruned, so pruning them broke the documented run. It is back to
its main version. The SageMaker Nova docstring now points at the file's
real location in tests/local_testing.
* test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit
Rename-only. Mirrors the old paths, including fixtures, the stubtest config
and the native-route wheel script. Two files that collide with existing unit
files are merged in a follow-up commit.
* test: merge, prune and repoint the moved core, routing, responses, caching and rust_bridge tests
Merges the two files that collided with existing unit files, folding the
legacy extra case into test_is_chat_completion_cached_dict, and deletes the
9 audited tests a stronger test in the same file already covers.
Keeps what needs the network in tests/test_litellm: test_tokenizers pulls a
tokenizer from the Hugging Face hub, and the gpt2 and r50k_base tokenizer
cases download their BPE files. The unit core_utils conftest points
TIKTOKEN_CACHE_DIR at litellm's bundled encodings so the rest never depend on
import order to stay offline, and FakeSecretVault moves to a shared module
so both trees can build it.
* ci: run the moved core, routing, responses, caching and rust_bridge tests under their flags
core_utils gets a core-utils flag and CircleCI job, and its GHA shard keeps
the legacy path for the retained network tests. router_utils and
router_strategy join enterprise-routing, responses joins
responses-caching-types (minus responses/mcp, which mcp-integration owns),
caching joins caching-local and rust_bridge joins misc. The redis-compat,
test-rust, stubtest and merge-smoke paths follow the move.
* docs: point the Rust crate references at tests/unit
* test: make the moved core, routing and rust_bridge directories packages
* test: keep the no-loop DualCache batch_get_cache regression test
It runs the sync path outside any event loop, which the inside-loop test
cannot, so a change that picks the Redis client by loop state would only
show up there.
* test: keep the job's UNIT_FLAG out of the shard-script tests
* fix(url_utils): block 192.0.0.0/24 on every Python patch release
* test: move the new budget limiter tests into tests/unit/router_strategy
* test: move the new sentry scrubbing tests into tests/unit/litellm_core_utils
* test: move the new zerobus tests into tests/unit/integrations
* test: make tests/unit/integrations/zerobus a package
* test: load litellm's own tiktoken cache setup once instead of resetting it per test
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): honor per-second custom pricing on chat completions for every provider
* test(cost): assert a per-second priced deployment bills instead of staying at $0
The zero-cost diagnostic tests from #42345 used a per-second-only entry as their
misconfigured fixture, which this branch now bills. Switch that fixture to a
per-query-only entry, which is still selected as the deployment's own pricing and
still prices chat usage at $0, and add a per-second test asserting the call
duration is billed with no diagnostic
Also let a caller's explicit total_time outrank the logging window in
completion_cost, so the SDK precedence stays stamped response, caller, logging
* test(response_metadata): move the per-second pricing regression into the mapped tests/unit file
* fix(cost_calculator): keep media-mode per-second rates off the wall-clock path
A video, transcription, speech, or realtime entry's per-second rates price media seconds, which
their dedicated cost paths bill from the media itself. The generic per-second branch now skips
those modes, so a video status poll on a per-second video model bills nothing instead of the
seconds the poll took to answer.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
