Register Sail (providers.json, LlmProviders.SAIL, OpenAI-compatible lists,
ProviderConfigManager) for chat, Responses and /v1/messages, and add its 12
models to both cost maps with asap, balanced and flex price columns.
Sail picks speed and price with metadata.completion_window and rejects
service_tier, so the Sail chat and Responses configs translate the tier:
default and priority to asap, flex to flex, balanced to balanced, auto to no
window. Billing prices the window that was sent. A tier Sail has no window
for, or a window or tier set where billing cannot see it (request metadata,
extra_body), is a 400 unless drop_params is set.
Add balanced to ServiceTier and its _balanced price columns to the model
info types, the Rust catalog and the dashboard schema. A transform_extra_body
hook on the chat and Responses base configs, which returns extra_body
unchanged by default, lets Sail keep the window when a caller also sends
extra_body.metadata. Sail is listed in the Add Model form and model picker.
Co-authored-by: shrey kharbanda <shrey@berri.ai>
* feat(proxy): add fail_closed_rate_limit_enforcement to reject requests with 503 while Redis rate limit counters are unreachable
* fix(proxy): reject fail-closed rate limit checks before logging the in-memory fallback and pin the boot warning in the lifespan
* fix(proxy): coerce the fail-closed flag, fail closed on read-only checks, and refund partial cluster increments
* fix(proxy): window-guard rate limit refunds and catch the fail-closed rejection by type
* fix(proxy): read the compaction rate-limit gate's limiter from the proxy hook registry
* fix(proxy): count the pending request in read-only rate-limit checks and keep the compaction gate off the caller's parallel slot
The compaction polyfill's summary-model gate, once it ran against the real v3 limiter, showed two behaviors nobody had chosen. The read-only check compared the stored counter with the same `>` the increment path uses, but a read-only check decides a request that has not been counted yet, so a summary model exactly at its rpm limit still went out. The read-only path now adds the pending increment of 1 before comparing; the increment path is unchanged.
The gate also passed the key's max_parallel_requests gauge through, and the read-only gauge count includes the caller's own in-flight slot, so a key with max_parallel_requests: 1 never compacted. The gate now drops that gauge from its descriptors, since the summary call runs inside a request the limiter already admitted.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(otel): detach post-response service spans by request phase, name redis spans by operation
Service spans logged from the post-response phase (success callbacks, the response-cache write) now root their own trace linked to the request span even while the server span is still recording, instead of only when they happen to end after it. Redis service spans are named `redis <operation>`; the litellm call chain that issued them moves to the `litellm.service.caller` attribute via a typed `ServiceLoggerPayload.caller` field.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep the service caller on failure and legacy spans, test the production phase dispatch sites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): mark anthropic messages stream cache write as post-response phase
The /v1/messages streaming cache writer awaits async_add_cache inline
instead of going through create_cache_write_task, so its redis span
stayed parented under the request trace. Wrap the write in
post_response_phase so it detaches like the chat completions write.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): write the Messages stream cache in a background task after handoff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts
The Langfuse redaction unit test exports to a local OTLP capture instead of
polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill
test only requires spend rows for requests the surviving worker served. The
spend-routes sweep treats the intentional /spend/capture_rate 503 as expected.
CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper
instead of reading it as "variable exists". The integration egress guard now
matches the script's own cgroup, so it no longer blocks the CircleCI agent,
which runs as the same user.
* fix(ci): keep the policy-load backoff typed as float
* fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests
* fix(secrets): load CyberArk policy one request at a time per manager
* test(secrets): pin that non-conflict CyberArk policy failures are not retried
* test(unit): run tests/unit with only an allowlisted host environment
CircleCI's unit job inherits every project env var, so real provider keys,
REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that
assume none are set. Locally, litellm's import-time load_dotenv did the same
from any .env up the tree. The unit conftest now drops every variable outside
a small allowlist and disables dotenv before litellm is imported.
* test(e2e): name a failed search and the stuck batch status instead of misattributing them
The websearch session test read an empty web_search_tool_result_error block as a
successful search, so a failing search tool surfaced as a session billing bug.
The batch cancellation timeout now reports the last status the proxy returned.
* fix(ci): scrub the host environment per unit test instead of for the whole pytest process
GHA shards run tests/unit next to other suites in one process, so the import-time
scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP
upstream fell back to the SDK2 interpreter. The two websearch tests that called
OpenAI and Perplexity live are removed: tests/unit no longer sees their keys.
* fix(ci): scrub only the host variables present before litellm is imported
The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to
its bundled encodings, so tokenizer paths tried to download them and hit the
socket guard. The prisma setup test now passes its own database URL instead of
reading one another test leaked into the process environment.
* fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue
LoggingWorker marked a task done on whichever queue was current when the callback
finished, so a callback that outlived an event-loop change raised "task_done()
called too many times" or undercounted the new loop's queue. It now settles the
queue the task came from.
The rest are test isolation fixes for failures that only appeared when another
file ran first on the same xdist worker: a replaced user_api_key_cache, breaker
metrics unregistered by prometheus tests, semantic_router's health-check filter on
uvicorn.access, logging tasks carried over from bedrock tests, a Router-written
model_cost entry, and a stray post captured by the langflow test. The token
counter check now asserts bounded chunking instead of wall-clock time.
* test(e2e/ui): wait for the logout redirect before visiting a protected page
Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED).
* test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests
* test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read
* test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect
* test(integration/mcp): answer the model-info probe another test's proxy sends to the model double
* test: move key-gated tests/test_litellm SDK tests into tests/llm_translation and drop empty folders
* test: make token counter and health check unit tests run offline
* ci: point unit shards, rust path filter, Makefile and docs at tests/unit
* docs: fix stale test_litellm run paths in moved llm_translation tests
* fix: correct databricks e2e sys.path depth and contributing example path
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: load the embedding base image from a committed 100x100 PNG instead of downloading it
* test: move the volcengine embedding test into tests/unit
* test: check gpt2 and r50k_base tokenizer parity against committed tiktoken reference files
* test: check hub tokenizer selection against an in-memory Hugging Face hub
* test: serve image URLs from respx in the gemini tool-result and format-param tests
* ci: drop the emptied legacy core-utils test path
* test: cover the cohere and anthropic tokenizer paths in the hub tokenizer test
* test: fetch every format-param image through respx and check its bytes reach the request
* test: drop the gpt2 and r50k_base parity tests, which no litellm path uses
* test: drop comments that restate assertions in the format-param test
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rename fork-flag to unit-flag now that it applies on every event
* test: move tests/test_litellm root and small trees into tests/unit
Pure renames, no content changes. Follow-up commits in this PR fix
references, merge the three files that already existed in tests/unit,
keep live-provider tests in tests/test_litellm and wire CI.
* test: carry tests/test_litellm conftest isolation into tests/unit
Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS,
proxy-URL and keychain env, and session-end client cleanup now reset for
unit tests too. The environment isolation owns its MonkeyPatch so a test's
own monkeypatch is undone before the model-cost teardown runs.
* test: merge, split and prune the moved root and small-tree tests
Merge batches/test_batch_utils.py and the chat_completions and messages
dispatch tests into the files that already existed in tests/unit. Keep
the live Gemini interactions tests, the async image-fetch format test and
the OpenAI embedding scorer test in tests/test_litellm since they need
real network or keys. Put test_router.py under tests/unit/test_router so
the existing package no longer shadows it. Delete eight tests the audit
found superseded by stronger ones kept in this move.
* ci: run the moved root and small-tree tests under their legacy flags
Add the misc and responses-caching-types flags to unit_selection.sh and
CircleCI, extend enterprise-routing and mcp-integration, and point the
legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest
and change classifier at the new paths.
* test: make the new tests/unit directories packages
tests/unit/test_package_layout.py requires every directory to carry an
__init__.py, and without one the moved and retained
test_litellm_responses_bridge.py modules collide on import.
* test: scope the unit socket block to tests/unit in shared sessions
The GHA shards collect the legacy test-path and the unit selection in one
pytest session. The unit conftest's loopback-only block leaked into legacy
modules that reach the network at import. The legacy conftest now lifts the
restriction at collect and setup time, and the unit conftest re-applies it
when collecting its own modules.
* test: move tests/test_litellm/llms into tests/unit/llms
Rename-only. Moves the provider tests and the fine-tuning fixtures they
load, mirroring the old paths. Follow-up commits merge, split and wire them.
* test: merge, split and prune the moved llms tests
Merges the Databricks chat transformation tests into the existing unit
file, keeps the tests that need real keys or the network in
tests/test_litellm, deletes the audited tests a stronger unit test
already covers, and points imports at tests.unit.llms.
* ci: run the moved llms tests under their legacy flags
The Vertex AI and All Other Providers shards keep their legacy test-path
for the retained files and add the llm-vertex-ai and llm-other-providers
unit selections. CircleCI gets matching unit jobs.
* test: make the tests/unit/llms directories packages
Adds __init__.py to the moved dirs and drops the legacy ones whose
directories no longer hold tests.
* test: drop script runners and path hacks the llms split left dangling
The __main__ runners in the split openai_like files and the Databricks e2e
runner called tests that now live in the other half of the split or were
deleted. The retained legacy halves also no longer need sys.path edits.
* test: give the shard-script tests their own GITHUB_OUTPUT
They only passed where the runner set it. The CircleCI unit job's env
allowlist drops it, so the script's redirect failed there.
* test: point the router and module-deletion checks at tests/unit
router_code_coverage and code_qa_check_tests only searched tests/test_litellm,
so the moved router tests no longer counted. The two silent-experiment tests
the audit deleted were the only direct callers of those methods; they are
replaced with tests that assert the forwarded shadow request and the
recursion guard.
* test: move tests/test_litellm integrations and secret_managers into tests/unit
Rename-only. Mirrors the old paths, including the directory conftests
and the prompt and JSON fixtures. Follow-up commits prune and wire them.
* test: prune and repoint the moved integrations tests
Deletes the 7 audited tests a stronger test in the same tree already
covers, imports the TLS sink helpers from their new conftest path, and
restores os.environ after each integrations test. Some presets write
OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the
legacy tree's test ordering that header leaked into the AgentOps tests.
* ci: run the moved integrations tests under their legacy flag
The integrations GHA shard and a new CircleCI job run the integrations
unit selection. secret_managers joins the misc selection.
* docs: point integrations and secret_managers references at tests/unit
* test: make the moved integrations directories packages
* test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path
The Databricks e2e file is a manual script whose main() calls the tests
that were pruned, so pruning them broke the documented run. It is back to
its main version. The SageMaker Nova docstring now points at the file's
real location in tests/local_testing.
* test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit
Rename-only. Mirrors the old paths, including fixtures, the stubtest config
and the native-route wheel script. Two files that collide with existing unit
files are merged in a follow-up commit.
* test: merge, prune and repoint the moved core, routing, responses, caching and rust_bridge tests
Merges the two files that collided with existing unit files, folding the
legacy extra case into test_is_chat_completion_cached_dict, and deletes the
9 audited tests a stronger test in the same file already covers.
Keeps what needs the network in tests/test_litellm: test_tokenizers pulls a
tokenizer from the Hugging Face hub, and the gpt2 and r50k_base tokenizer
cases download their BPE files. The unit core_utils conftest points
TIKTOKEN_CACHE_DIR at litellm's bundled encodings so the rest never depend on
import order to stay offline, and FakeSecretVault moves to a shared module
so both trees can build it.
* ci: run the moved core, routing, responses, caching and rust_bridge tests under their flags
core_utils gets a core-utils flag and CircleCI job, and its GHA shard keeps
the legacy path for the retained network tests. router_utils and
router_strategy join enterprise-routing, responses joins
responses-caching-types (minus responses/mcp, which mcp-integration owns),
caching joins caching-local and rust_bridge joins misc. The redis-compat,
test-rust, stubtest and merge-smoke paths follow the move.
* docs: point the Rust crate references at tests/unit
* test: make the moved core, routing and rust_bridge directories packages
* test: keep the no-loop DualCache batch_get_cache regression test
It runs the sync path outside any event loop, which the inside-loop test
cannot, so a change that picks the Redis client by loop state would only
show up there.
* test: keep the job's UNIT_FLAG out of the shard-script tests
* fix(url_utils): block 192.0.0.0/24 on every Python patch release
* test: move the new budget limiter tests into tests/unit/router_strategy
* test: move the new sentry scrubbing tests into tests/unit/litellm_core_utils
* test: move the new zerobus tests into tests/unit/integrations
* test: make tests/unit/integrations/zerobus a package
* test: load litellm's own tiktoken cache setup once instead of resetting it per test
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(xai): add native xAI batches and files support
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(xai): tighten batch handler typing and avoid Final redeclaration on star import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(xai): walk batch result pages iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(xai): stop paging on empty pagination token and honor litellm.xai_key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(xai): import NotRequired and TypedDict from typing_extensions for Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(batches): accept image and video endpoints on batch create
* test(xai): lock batch endpoint, auth, and result contracts
The batches test package collided with litellm/batches under pytest prepend, so the All Other Providers shard could not collect the new tests.
* fix(xai): price grok batch usage at xAI's 20 percent batch discount
* fix(xai): map not-found file reads to 404, bill batch reasoning tokens, and add 200k batch tier rates
* refactor(xai): drop routine prose and move tests under tests/unit
* fix(health): hand the resolved provider to list_batches in batch-mode health checks
* test(xai): make tests/unit/llms/xai/batches a package
* fix(xai): walk every page of the files list by pagination_token
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): map legacy bucket_name to gcs_bucket_name and add GCS_BATCH_BUCKET_NAME env var
* refactor(router): keep legacy bucket_name as a credential field instead of a validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): pass the RAG corpus bucket to the file upload instead of hopping through GCS_BUCKET_NAME
* fix(vertex_ai): accept existing_file_id in the RAG Engine store step so ingest() runs end to end
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): surface a converse-stream 200 that decodes to no events as a 502 instead of an empty turn
* fix(bedrock): quote the body head only when a stream decoded no events
The leftover-bytes error keeps the byte and event counts, the content type and the request id but no longer quotes the first bytes of a stream that already decoded events, since that head is the start of a healthy stream and can hold model output. The anthropic_messages empty-stream warning no longer prints the request's model name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rename fork-flag to unit-flag now that it applies on every event
* test: move tests/test_litellm root and small trees into tests/unit
Pure renames, no content changes. Follow-up commits in this PR fix
references, merge the three files that already existed in tests/unit,
keep live-provider tests in tests/test_litellm and wire CI.
* test: carry tests/test_litellm conftest isolation into tests/unit
Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS,
proxy-URL and keychain env, and session-end client cleanup now reset for
unit tests too. The environment isolation owns its MonkeyPatch so a test's
own monkeypatch is undone before the model-cost teardown runs.
* test: merge, split and prune the moved root and small-tree tests
Merge batches/test_batch_utils.py and the chat_completions and messages
dispatch tests into the files that already existed in tests/unit. Keep
the live Gemini interactions tests, the async image-fetch format test and
the OpenAI embedding scorer test in tests/test_litellm since they need
real network or keys. Put test_router.py under tests/unit/test_router so
the existing package no longer shadows it. Delete eight tests the audit
found superseded by stronger ones kept in this move.
* ci: run the moved root and small-tree tests under their legacy flags
Add the misc and responses-caching-types flags to unit_selection.sh and
CircleCI, extend enterprise-routing and mcp-integration, and point the
legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest
and change classifier at the new paths.
* test: make the new tests/unit directories packages
tests/unit/test_package_layout.py requires every directory to carry an
__init__.py, and without one the moved and retained
test_litellm_responses_bridge.py modules collide on import.
* test: scope the unit socket block to tests/unit in shared sessions
The GHA shards collect the legacy test-path and the unit selection in one
pytest session. The unit conftest's loopback-only block leaked into legacy
modules that reach the network at import. The legacy conftest now lifts the
restriction at collect and setup time, and the unit conftest re-applies it
when collecting its own modules.
* test: move tests/test_litellm/llms into tests/unit/llms
Rename-only. Moves the provider tests and the fine-tuning fixtures they
load, mirroring the old paths. Follow-up commits merge, split and wire them.
* test: merge, split and prune the moved llms tests
Merges the Databricks chat transformation tests into the existing unit
file, keeps the tests that need real keys or the network in
tests/test_litellm, deletes the audited tests a stronger unit test
already covers, and points imports at tests.unit.llms.
* ci: run the moved llms tests under their legacy flags
The Vertex AI and All Other Providers shards keep their legacy test-path
for the retained files and add the llm-vertex-ai and llm-other-providers
unit selections. CircleCI gets matching unit jobs.
* test: make the tests/unit/llms directories packages
Adds __init__.py to the moved dirs and drops the legacy ones whose
directories no longer hold tests.
* test: drop script runners and path hacks the llms split left dangling
The __main__ runners in the split openai_like files and the Databricks e2e
runner called tests that now live in the other half of the split or were
deleted. The retained legacy halves also no longer need sys.path edits.
* test: give the shard-script tests their own GITHUB_OUTPUT
They only passed where the runner set it. The CircleCI unit job's env
allowlist drops it, so the script's redirect failed there.
* test: point the router and module-deletion checks at tests/unit
router_code_coverage and code_qa_check_tests only searched tests/test_litellm,
so the moved router tests no longer counted. The two silent-experiment tests
the audit deleted were the only direct callers of those methods; they are
replaced with tests that assert the forwarded shadow request and the
recursion guard.
* test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path
The Databricks e2e file is a manual script whose main() calls the tests
that were pruned, so pruning them broke the documented run. It is back to
its main version. The SageMaker Nova docstring now points at the file's
real location in tests/local_testing.
* test: keep the job's UNIT_FLAG out of the shard-script tests
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(anthropic): placement policy for mid-conversation system messages
Pure functions over the OpenAI-format message list: split off the leading
system run, keep later system messages as role=system at a placement Anthropic
accepts on models flagged supports_mid_conversation_system (after a user turn,
before an assistant turn or the end, never adjacent), and convert them to user
turns in place elsewhere, keeping tool_result first in a merged user turn.
* fix(anthropic): keep mid-conversation system out of the chat completions system prompt
translate_system_message hoisted every role=system message, at any index, into
the top-level system block. On a conversation carrying a mid-session reminder
that rewrites the cached prefix, so the provider re-bills the whole history at
cache-write pricing on every turn (#36559). #36968 fixed this on /v1/messages;
the chat completions path, shared by first-party Anthropic, Vertex, Azure AI
and Bedrock Invoke, still hoisted.
Only the leading system run becomes the system prompt now. Later system
messages go through the placement policy, and anthropic_messages_pt emits a
system message instead of rejecting the role. The caller's message list is no
longer mutated. Tests pin the two-turn prefix invariant across all four chat
configs and both flag states.
* refactor(anthropic): single-source the converted system note
The /v1/messages pass-through and the chat completions path must prefix a
converted system turn with the same operator note.
* test(e2e): prove the prompt cache survives a mid-conversation system reminder on chat completions
Same priming and assertions as the /v1/messages cases, through
/v1/chat/completions with OpenAI-format messages, for first-party Anthropic
and Bedrock Invoke on a flagged (Opus 4.8) and an unflagged (Haiku 4.5) model.
The reminder sits between the assistant turn and the next user turn, the shape
OpenAI-style agent frameworks send, which is the placement the chat path has
to translate.
* test(anthropic): cover the cache_control rebuild shapes and type the test helpers
Codecov flagged the 5m ttl branch and the empty-system path of the wire
builder; both now have a test. Greptile asked for full typing on the new
test helpers.
* refactor(anthropic): read the mid-conversation flag through a public supports_ helper
supports_mid_conversation_system joins the other supports_* helpers in
litellm.utils, so the chat transformation stops importing the private
_supports_factory.
* chore(typing): declare the mid-conversation type aliases with TypeAlias
The Final sweep tightened LIT010, which exempts TypeAlias declarations but
counts a bare alias assignment as an unannotated binding.
* fix(anthropic): let add_code_execution_tool take the pass-through message union
The translator now emits role=system inside messages for models that accept it,
so anthropic_messages_pt returns the pass-through union. add_code_execution_tool
still declared the narrower user/assistant union while only ever reading
content, so upstream's strip_advisor_blocks_from_messages call in between made
the mismatch visible to the type checker.
* fix(bedrock): keep mid-conversation system messages in place on converse path
* fix: ruff format + multi tool_result order + regression test
* fix: satisfy type-discipline gate + update osv ignore for mlflow PYSEC-2026-3865
* fix(bedrock): restore role narrowing in hoisted system loop for basedpyright budget
* test(bedrock): cover mid-conversation system conversion branches
- non-dict guard in _opens_with_tool_result
- in-place conversion without tool context
- str/list cache_control preservation in mid-conversation path
- drop unreachable non-system guard in hoisted loop
* Place type-discipline suppressions on the lines the gate scans
* Narrow hoisted loop to system role so basedpyright sees the right TypedDict
* fix(anthropic): place mid-conversation system runs by their neighbours only
A run after an assistant turn now slides behind the user turn that
immediately follows it, and a run that ends the array or precedes an
assistant turn becomes a user turn in place. No later message can move
an earlier run, so a client that replays the conversation with more
turns appended sends a byte-identical prefix and preserved thinking
blocks keep their binding
* refactor(bedrock): share the converted system note with the anthropic module
Converse imports CONVERTED_SYSTEM_NOTE instead of carrying its own copy
of the same text, and the reordering helpers lose their comments
* test: pin the replayed request prefix across preserved-thinking turns
One test per audited feature, through the real entrypoint: the chat
transformations for anthropic, bedrock invoke, vertex and converse, the
modify_params dummy tool result, dotprompt with unchanged variables, and
Presidio masking against an in-process fake. Each serializes system,
tools and the earlier messages of turn N and N+1 and asserts they match.
The e2e mid-conversation system test imports its content blocks from
models.py again and is marked provider_live
* fix(anthropic): move mid-conversation system placement into prompt_templates
The prompt factory imported the placement helper from the Anthropic provider
package, whose common_utils reads a factory constant at import time, so loading
the factory first raised ImportError. The module now sits next to
anthropic_messages_pt and every consumer imports core utils
A user turn with content [] or None puts no block on the wire, so a system run
anchored to it landed first in messages or behind an assistant turn. Such a run
now converts in place; empty strings and empty text blocks still anchor because
the factory fills them with a placeholder
* fix(anthropic): anchor system messages only on user turns that reach the wire
* fix(bedrock): type the converse system-message helpers over the message TypedDicts
* fix(anthropic): read replayed pydantic messages in the Converse helpers and convert a system run whose assistant follower sends nothing
A history that replays the previous turn as the litellm.Message object
was invisible to the Converse system-message helpers, so a mid-conversation
system stayed between a tool call and its result or reached Converse as
role: system. The helpers now read fields through the shared
message_field and parts_of accessors and drop the local role predicate.
Flagged placement anchored a system run on any assistant follower, but
anthropic_messages_pt drops an assistant turn that puts no block on the
wire (content None, an empty list, an unsigned thinking part), so the
system landed directly before the next user turn, which Anthropic
rejects. Such a run now converts in place. An empty or whitespace text
turn still anchors, since the converter pads it with a placeholder.
* fix(anthropic): treat bridged encrypted reasoning as a vanishing assistant turn for system placement
An assistant turn whose only blocks carry Responses API encrypted reasoning is
dropped by anthropic_messages_pt, so a mid-conversation system run anchored
before it landed directly before the next user turn. The unsignable-thinking
predicate now lives in common_utils and both the factory and the placement
policy consult it.
* fix(anthropic): let an inline thinking part hide separate thinking_blocks in system placement
anthropic_messages_pt skips an assistant turn's separate thinking_blocks as soon
as its content list carries an inline thinking or redacted_thinking part, so a
turn whose inline part is unsigned puts nothing on the wire even when the
separate block is signed. The placement policy now mirrors that rule.
---------
Co-authored-by: Shifat Islam Santo <shifatislamsanto764@gmail.com>
Co-authored-by: ege-arhan <egearhany@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): stop advertising OpenAI platform-only params on Gemma and Llama routes
The Anthropic /v1/messages bridge derives prompt_cache_key from Claude Code's
session id whenever the provider config advertises it, and every Vertex
OpenAI-compatible route (gemma/, openai/<endpoint>, meta/) inherited the full
OpenAI list, so the Model Garden vLLM container rejected each turn with a
pydantic extra_forbidden 400. Vertex's Llama and Gemma configs now filter one
shared list of platform-only params (prompt_cache_key, prompt_cache_retention,
safety_identifier, service_tier, store, web_search_options, modalities,
prediction, audio, max_retries) out of their supported params, so the bridge
no longer derives the key and drop_params drops an explicit one.
* fix(vertex_ai): scope the platform-param filter to self-deployed Model Garden endpoints
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): surface the Gemma container's own error inside a 200 :predict response
* refactor(vertex_ai): move the gemma container error parser next to its adapter
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token
A CLI session token is a fresh random secret on every login, so since v1.99 each
login's spend rows carried a different sha256 hash as api_key and the usage APIs
could resolve neither key_alias nor user_email for them. Spend rows and logging
callbacks now attribute a session request to its stable alias,
cli-session-<user_id>, and the usage endpoints derive that alias and owner from
the key itself instead of scanning for a matching digest
* fix(spend): resolve the CLI session team from the user's first team in usage metadata
A cli-session key carries no team of its own in the DB, so the usage
breakdown showed team_id None for it and the export grouped it as
Unassigned. The login attaches the user's first team to the session, so
the recovery mirrors that rule for cli-session keys only.
* fix(spend): claim the session team only for a single-team user
The CLI login attaches a team on its own only when the user has exactly
one; a user in several teams picks one per login, so usage metadata for
the alias would otherwise name a team the login may not have used.
* test(pass_through): mark the mocked auth object as a plain key
The logged key follows the alias only for a session token; a bare
MagicMock reads as one, so the test names the field it relies on.
* fix(spend): attribute CLI session pass-through, queue, and managed batch spend to the cli-session alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): only treat the exact cli-session-<created_by> value as a batch key alias
A managed object row written by an older build can still carry the raw per-login
session token, which shares the cli-session- prefix. Matching on the prefix alone
would have surfaced that token as a trusted alias and persisted it verbatim in the
batch cost spend log, so the alias check now requires the exact per-user value and
every other prefixed value keeps going through redaction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): log proxy executed batch rows under the cli-session alias instead of the session token
_row_metadata set user_api_key from the raw bearer token while user_api_key_hash carried the alias, so the spend log redaction rejected the alias as untrusted and hashed the random session token instead
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): attribute semantic search embedding spend to the cli-session alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): scope /key/spend/report for a CLI session to the cli-session alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): use the cli-session alias for websearch spend, prometheus failure labels and the parallel limiter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(spend): drop explanatory docstrings on get_logged_api_key and attach_user_details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): only recover cli-session usage keys whose suffix is a known user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ocr): remove the Python OCR execution path and require the Rust route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fmt
* refactor(ocr): tidy the native OCR passthrough binding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ocr): ruff format the azure passthrough transformation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ocr): resolve passthrough OCR costing in one Rust call
Replace passthrough_url/passthrough_transform with passthrough_response,
which matches the relayed endpoint against each Azure config's path
segments instead of building a fake request to call get_complete_url.
The binding drops the unused headers, status and api_base arguments.
Catch the ValueError/RuntimeError the binding raises so a relayed body
that is not OCR-shaped falls back to the passthrough object instead of
failing logging, and cover the relay against the real binding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ocr): drop the unused LlmProviders import from health check helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: drop the ocr_testing job now that tests/ocr_tests is gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(ocr): restore the live OCR matrix and the ocr_testing job
The public litellm.ocr / aocr / Router interface is unchanged by the Rust
migration, so the live provider matrix still applies. Drops the stale VCR skip
list for the deleted test_rust_bridge.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(ocr): import Final in the health check helper tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): map Anthropic batch row params the way real time does
* fix(bedrock): let a batch row's allowed_openai_params reach the mapper
* test(bedrock): assert the batch thinking value matches the real-time mapping
* fix(bedrock): keep json_mode out of Anthropic batch rows and pin route-prefixed deployments
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge
Vertex batch uploads treated every non-embeddings JSONL row as a chat
completions body, so a /v1/responses row lost its input and reached GCS
as a blank text part. Route detection now recognizes /v1/responses rows
and bridges them to chat through the same Responses-to-Chat bridge the
real-time path uses. That bridge call moves out of the Bedrock files
transformation into a shared helper both providers call, forwarding the
record's fields as sent, like real time, instead of validating them
against the SDK TypedDicts whose required keys clients omit.
* chore(batches): type the Vertex responses test helper and drop the quoted input cast
* fix(batches): translate developer messages to system on Vertex and Bedrock batch rows like real time
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(logging): pass provider response headers to callbacks on every endpoint
Custom callbacks only received kwargs["response_headers"] for chat
completions. Responses, image generation and edit, speech, and
transcription calls either never recorded the provider's headers or
recorded them in one place and not the other.
Every handler now records the provider's httpx headers on the response's
hidden params as "headers" (raw) and "additional_headers" (processed,
with LiteLLM's own entries winning on a clash), and the logging object
derives model_call_details["response_headers"] from those hidden params
before cost calculation on the non-stream and both streaming success
paths, keeping a handler-set value authoritative. Binary speech responses
expose their hidden params to the standard logging payload, and the sync
OpenAI transcription request always fetches the raw response.
* test(images): point the legacy image and speech fakes at the raw response surface
Image generation now goes through the SDK's raw response so the provider headers can be read, and the speech binary response now carries hidden params. The unit fakes in the image generation, xinference, proxy provider, image edit, Vertex speech, and otel suites still pinned the old call surface and the old "no hidden params" assertion, so they read an uncalled mock or a fake response without headers.
* test(images): drop the rewritten mock comments and the generated edit PNGs
* test(images): move the llm-span test's image fake to the raw response surface
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The new CircleCI tests pipeline (#42773) runs tests/unit under pytest-cov on
CPython 3.12.2, where coverage traces every line through sys.settrace. The two
tracemalloc peak comparisons in test_vertex_ai_files_streaming.py drive 8000-row
payloads through both pipelines and slow from ~10s to over 3 minutes under that
tracer, so both hit the 90s pytest-timeout on every run.
Mark them no_cover so pytest-cov pauses tracing for just these two. Their
assertions are unchanged and every other test in the file still reports coverage.
* fix(fireworks_ai): route firerouter short names and bill pass-through legs at the routed model's rates
fireworks_ai/firerouter and fireworks_ai/firerouter/<slug> resolve to
accounts/fireworks/routers/... instead of a models/ path, and the cost
calculator falls back to the routed model's own catalog entry before the
Fireworks size buckets so a Claude leg is no longer priced at $0
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fireworks_ai): bill routed legs under the routed model's own provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fireworks_ai): require the k suffix when parsing tiered input fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(params): carry stream_chunk_size through litellm_params instead of provider params
* test(integration): fence stream_chunk_size out of every provider request body
* test(bedrock): type parametrized stream chunk test params
* test(integration): drop the contracts manifest resurrected by the main merge
* test(bedrock): type the stream_chunk_size test helpers
* test(params): finish AGENTS.md typing pass on stream_chunk_size tests
* test(integration): drop the covers marker from the stream_chunk_size wire test
---------
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
* fix(bedrock): send json_schema as a forced tool on Claude Opus 4.7 and 4.8 Converse
Bedrock rejects outputConfig.textFormat on Opus 4.7 and 4.8 with
"output_config.format: Extra inputs are not permitted", and the AWS
model cards list structured outputs as not supported for both, so
their cost-map entries no longer claim supports_native_structured_output
and json_schema requests fall back to the json_tool_call tool.
Fixes#27846
* test(bedrock): assert Opus 4.7 and 4.8 inline the schema on Invoke, move the native case to Sonnet 4.6
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(cost-map): remove models past their deprecation date
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop the empty parametrize left behind by the gemini web search removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): drop merge base block left by conflict resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop gemini image cost tests pinned on removed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make bedrock collector and secret scan timing tests deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): count interpreter calls instead of wall clock in the secret scan scaling test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): profile the secret scan with cProfile, restore the outer profiler and tighten the scaling bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep S3 signer test additions scoped to new cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): drop e2e suite changes from the S3 signing fix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): build S3 credentials directly from the s3_* pair so ambient AWS_* env never mixes in
Restores the split-identity e2e coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
