mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
124 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yassin
|
f46ddd24aa |
feat(proxy-extras): make the SpendLogs index build opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
24e0d53fc2 |
fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948)
The two SpendLogs index migrations shipped in v1.103.0 each break one table shape: the plain CREATE INDEX holds a SHARE lock on a large unpartitioned table and the CONCURRENTLY one fails with 0A000 on a partitioned parent. Both files are now inert and the indexes are built by a table-driven, shape-aware step after migrate deploy: CONCURRENTLY on a plain table, ON ONLY the parent plus per-partition CONCURRENTLY and ATTACH PARTITION on a partitioned one. The migration job builds them synchronously and exits non-zero on failure; a serving proxy that ran migrate deploy itself builds them in the background off the readiness path. A valid index of the same definition under another name is renamed and reused, an invalid one is rebuilt, and extra copies are reported with their DROP INDEX statement instead of being dropped. The migration checker rejects any CREATE INDEX on LiteLLM_SpendLogs or LiteLLM_ErrorLogs in future migrations
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit
|
||
|
yuneng-jiang
|
1df3f10641
|
test(unit): stop test modules from putting their own directory on sys.path (#43420)
test_ssl_verify_unit.py inserted tests/unit at the front of sys.path, so any later import of litellm_proxy_extras resolved to the tests/unit/litellm_proxy_extras test package. Whenever the CircleCI shard split collected that file before test_litellm_proxy_extras_logging.py, collection failed with ModuleNotFoundError. test_gemini_session_leak.py had the same insert for its own directory |
||
|
yuneng-jiang
|
e0cbb0db82
|
fix(streaming): keep litellm Usage on text-completion usage chunks (#43047) (#43402)
* fix(streaming): keep litellm Usage on text-completion usage chunks
* fix(streaming): convert provider usage to litellm Usage instead of dropping it
(cherry picked from commit
|
||
|
yuneng-jiang
|
95f33f477d
|
chore: backport CircleCI speedups (#43347) and cost-map fixes (#42951) to rc/1.104.0 (#43366)
* fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) (cherry picked from commit |
||
|
yuneng-jiang
|
822ccfc11d
|
fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) (#43354)
* fix(s3_v2): drop terminal upload failures, bound retries per flush and enforce the queue cap at enqueue
* fix(s3_v2): keep retrying credential-rotation 403s, only AccessDenied-style errors are terminal
* test(env_keys): exclude DEFAULT_S3_MAX_FLUSH_ATTEMPTS as an internal tuning var
* fix(s3_v2): retry every 5xx, warn on first queue overflow, validate the flush budget
* fix(s3_v2): read the queue cap defensively so un-initialized loggers still enqueue
* fix(s3_v2): drop the getattr in _enqueue and tighten the retry tests
* fix(s3_v2): keep the constructor flush budget when the callback override is invalid
* feat(s3_v2): adapt per-object upload concurrency to sink latency and throttling
* refactor(s3_v2): tidy adaptive limiter
* perf(s3_v2): wake one waiter per released upload slot
* feat(s3_v2): make the enqueue queue cap configurable with s3_max_queue_size
* test(s3_v2): audit cells for cache hits, coded 403 and callback modes
* fix(s3_v2): retry bucket-wide failures by default, age-budget requeues and make terminal drops and adaptive concurrency opt-in
* refactor(s3_v2): suppress the missing-waiter ValueError explicitly in the adaptive limiter
* fix(s3_v2): count oldest events trimmed after a failed flush as callback failures
* style(s3_v2): move the mutable-ok marker onto the list literal it suppresses
* fix(s3_v2): report post-flush overflow drops once and grow adaptive concurrency above the floor before asserting back-off
* test(s3_v2): fail the SlowDown back-off test when the measured window sees no PUTs
* fix(s3_v2): restore base retry defaults, opt-in age budget, no enqueue cap, back off outside the limiter slot
* fix(s3_v2): hoist the default no-op upload slot to a module constant
* fix(s3_v2): drop unused mutable-ok suppressions on queue appends
* fix(s3_v2): keep the retry queue oldest-first and prioritise fresh events at upload time
* style(s3_v2): keep the mutable-ok marker on the queue list literal
* fix(s3_v2): build request bodies inside the upload slot and keep the sync retry set at base parity
* test(s3_v2): drop wall-clock sleeps from the unit tests
* fix(s3_v2): rebuild the request body inside the slot on every retry attempt
* test(s3_v2): anchor the backoff window on the first observed failure and tighten shard assertions
* fix(s3_v2): default the upload slot to the logger limiter so monkeypatched doubles keep working
* test(s3_v2): clear ambient AWS env credentials so the rotating profile signs the sync retry test
* test(s3_v2): shrink the linear send-batch perf test to 2k/8k elements
* test(s3_v2): fix stale batch sizes in the perf test assert message
* fix(s3_v2): keep async in-call retries on the base 403/500/503 set
* fix(s3_v2): hold the upload slot across retries, restore the bool upload contract, and fail safe on bool config
* fix(s3_v2): mark dropped uploads by element identity so a shared key cannot mask a retryable sibling
* fix(s3_v2): make the per-flush drop lookup constant time
* fix(s3_v2): take the upload slot in the caller like base, build the body once per attempt loop
* fix(s3_v2): match base retry, logging and hook behaviour unless the new options are opted in
* test(s3_v2): assert the wire key in the init-bypassed sync upload test
* test(s3_v2): assert the signed headers and wire key in the init-bypassed sync upload test
* fix(s3_v2): default the upload limiter at class level instead of reading it with getattr
* fix(s3_v2): drop the duplicate annotations that redeclare the class-level counters
* feat(s3_v2): drop terminal-failed uploads by default and bound retry age to one hour
* fix(s3_v2): fall back to the configured retry age on invalid values
* test(s3_v2): drive retry-age tests from a fixed clock
* test(s3_v2): audit cells for retry-age opt-out and 429 single-put parity
---------
(cherry picked from commit
|
||
|
yuneng-jiang
|
7248fd8328
|
chore: backport CI fixes, #43206 and Rust changes to rc/1.104.0 (#43338)
* feat(model-catalog): add Rust registry validation (#43136) * doc * feat(model-catalog): add Rust registry validation * test(model-catalog): ignore integration tests * docs(model-catalog): fix validation note grammar --------- Co-authored-by: Yujong Lee <yujong@berri.ai> (cherry picked from commit |
||
|
yuneng-jiang
|
4b3c72ac0e
|
test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) (#43267)
test_update_config_success_callback_normalization replaced
proxy_server.proxy_logging_obj with a MagicMock and never restored it.
Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping,
end-user and MCP tests on the same xdist worker awaited that mock and
failed. The test now uses monkeypatch.
test_prometheus_logging_callbacks set verbose_logger to DEBUG and
litellm.set_verbose at import, so every worker in the unit job ran with
DEBUG on. That broke caplog equality in the JEV classifier test, the
vertex streaming memory ratio, and four event-loop lag checks. The
module-level setup is removed; nothing in the file depended on it.
#43081 removed the OCR harness modules but left them in the
importability parametrize list.
test_get_model_info_bedrock_region reassigned litellm.model_cost and set
LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared
the get_model_info caches, so it failed whenever an earlier test had
looked up the regional model. It now uses monkeypatch and invalidates
the caches; the local_testing isolation fixture also invalidates them
after restoring model_cost.
The Windows job hit CircleCI's 10 minute no-output limit while cargo
compiles the Rust crates inside uv sync and uv build. Those two steps now
allow 30 minutes of silence.
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
ccf866c801
|
fix(router): await budget redis pipeline before sync reads (internal copy of #32618) (#43125)
* fix(router): await budget redis pipeline before sync reads * refactor(router): remove superseded Redis flush helper * fix(router): preserve concurrent spend during Redis synchronization * fix(router): type per-key spend totals without loop Final bindings * fix(router): log Redis failures before cancellable cleanup * fix(router): finalize Redis batches before propagating cancellation * test(router): reproduce cancellation while Redis cleanup is blocked * test: align budget hotpath checks with awaited Redis flush * fix(budgets): finish Redis flush after cancellation while queued * test(budgets): consolidate Redis regressions in mapped tests * fix(budgets): coalesce failed Redis increments by key * test(budgets): assert spend behavior instead of batch state * perf(router): sum pending budget spend by key once per sync --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
020e5dee9b
|
fix(anthropic): keep the replayed prefix byte-stable for preserved thinking on chat completions (#42630)
* feat(anthropic): placement policy for mid-conversation system messages Pure functions over the OpenAI-format message list: split off the leading system run, keep later system messages as role=system at a placement Anthropic accepts on models flagged supports_mid_conversation_system (after a user turn, before an assistant turn or the end, never adjacent), and convert them to user turns in place elsewhere, keeping tool_result first in a merged user turn. * fix(anthropic): keep mid-conversation system out of the chat completions system prompt translate_system_message hoisted every role=system message, at any index, into the top-level system block. On a conversation carrying a mid-session reminder that rewrites the cached prefix, so the provider re-bills the whole history at cache-write pricing on every turn (#36559). #36968 fixed this on /v1/messages; the chat completions path, shared by first-party Anthropic, Vertex, Azure AI and Bedrock Invoke, still hoisted. Only the leading system run becomes the system prompt now. Later system messages go through the placement policy, and anthropic_messages_pt emits a system message instead of rejecting the role. The caller's message list is no longer mutated. Tests pin the two-turn prefix invariant across all four chat configs and both flag states. * refactor(anthropic): single-source the converted system note The /v1/messages pass-through and the chat completions path must prefix a converted system turn with the same operator note. * test(e2e): prove the prompt cache survives a mid-conversation system reminder on chat completions Same priming and assertions as the /v1/messages cases, through /v1/chat/completions with OpenAI-format messages, for first-party Anthropic and Bedrock Invoke on a flagged (Opus 4.8) and an unflagged (Haiku 4.5) model. The reminder sits between the assistant turn and the next user turn, the shape OpenAI-style agent frameworks send, which is the placement the chat path has to translate. * test(anthropic): cover the cache_control rebuild shapes and type the test helpers Codecov flagged the 5m ttl branch and the empty-system path of the wire builder; both now have a test. Greptile asked for full typing on the new test helpers. * refactor(anthropic): read the mid-conversation flag through a public supports_ helper supports_mid_conversation_system joins the other supports_* helpers in litellm.utils, so the chat transformation stops importing the private _supports_factory. * chore(typing): declare the mid-conversation type aliases with TypeAlias The Final sweep tightened LIT010, which exempts TypeAlias declarations but counts a bare alias assignment as an unannotated binding. * fix(anthropic): let add_code_execution_tool take the pass-through message union The translator now emits role=system inside messages for models that accept it, so anthropic_messages_pt returns the pass-through union. add_code_execution_tool still declared the narrower user/assistant union while only ever reading content, so upstream's strip_advisor_blocks_from_messages call in between made the mismatch visible to the type checker. * fix(bedrock): keep mid-conversation system messages in place on converse path * fix: ruff format + multi tool_result order + regression test * fix: satisfy type-discipline gate + update osv ignore for mlflow PYSEC-2026-3865 * fix(bedrock): restore role narrowing in hoisted system loop for basedpyright budget * test(bedrock): cover mid-conversation system conversion branches - non-dict guard in _opens_with_tool_result - in-place conversion without tool context - str/list cache_control preservation in mid-conversation path - drop unreachable non-system guard in hoisted loop * Place type-discipline suppressions on the lines the gate scans * Narrow hoisted loop to system role so basedpyright sees the right TypedDict * fix(anthropic): place mid-conversation system runs by their neighbours only A run after an assistant turn now slides behind the user turn that immediately follows it, and a run that ends the array or precedes an assistant turn becomes a user turn in place. No later message can move an earlier run, so a client that replays the conversation with more turns appended sends a byte-identical prefix and preserved thinking blocks keep their binding * refactor(bedrock): share the converted system note with the anthropic module Converse imports CONVERTED_SYSTEM_NOTE instead of carrying its own copy of the same text, and the reordering helpers lose their comments * test: pin the replayed request prefix across preserved-thinking turns One test per audited feature, through the real entrypoint: the chat transformations for anthropic, bedrock invoke, vertex and converse, the modify_params dummy tool result, dotprompt with unchanged variables, and Presidio masking against an in-process fake. Each serializes system, tools and the earlier messages of turn N and N+1 and asserts they match. The e2e mid-conversation system test imports its content blocks from models.py again and is marked provider_live * fix(anthropic): move mid-conversation system placement into prompt_templates The prompt factory imported the placement helper from the Anthropic provider package, whose common_utils reads a factory constant at import time, so loading the factory first raised ImportError. The module now sits next to anthropic_messages_pt and every consumer imports core utils A user turn with content [] or None puts no block on the wire, so a system run anchored to it landed first in messages or behind an assistant turn. Such a run now converts in place; empty strings and empty text blocks still anchor because the factory fills them with a placeholder * fix(anthropic): anchor system messages only on user turns that reach the wire * fix(bedrock): type the converse system-message helpers over the message TypedDicts * fix(anthropic): read replayed pydantic messages in the Converse helpers and convert a system run whose assistant follower sends nothing A history that replays the previous turn as the litellm.Message object was invisible to the Converse system-message helpers, so a mid-conversation system stayed between a tool call and its result or reached Converse as role: system. The helpers now read fields through the shared message_field and parts_of accessors and drop the local role predicate. Flagged placement anchored a system run on any assistant follower, but anthropic_messages_pt drops an assistant turn that puts no block on the wire (content None, an empty list, an unsigned thinking part), so the system landed directly before the next user turn, which Anthropic rejects. Such a run now converts in place. An empty or whitespace text turn still anchors, since the converter pads it with a placeholder. * fix(anthropic): treat bridged encrypted reasoning as a vanishing assistant turn for system placement An assistant turn whose only blocks carry Responses API encrypted reasoning is dropped by anthropic_messages_pt, so a mid-conversation system run anchored before it landed directly before the next user turn. The unsignable-thinking predicate now lives in common_utils and both the factory and the placement policy consult it. * fix(anthropic): let an inline thinking part hide separate thinking_blocks in system placement anthropic_messages_pt skips an assistant turn's separate thinking_blocks as soon as its content list carries an inline thinking or redacted_thinking part, so a turn whose inline part is unsigned puts nothing on the wire even when the separate block is signed. The placement policy now mirrors that rule. --------- Co-authored-by: Shifat Islam Santo <shifatislamsanto764@gmail.com> Co-authored-by: ege-arhan <egearhany@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f61b3c3f38
|
refactor(types): declare litellm-owned kwargs as typed objects and derive the lists from their fields (#42843)
* refactor(types): declare litellm-owned params in one registry * refactor(types): re-export registry constants without redundant aliases * refactor(types): satisfy type-discipline rules in registry projections and tests * refactor(types): classify every registry entry and check groups against typed config models * test(types): pin load-bearing names and exact projections in registry tests * style(types): keep agentic projection comment within ruff format * refactor(types): declare litellm-owned params as typed objects and derive the lists from their fields * refactor(types): fields of the typed objects become the registry; tests use a hand-written inventory * refactor(types): split traversal into wire_names and owned_wire_names, move rust to kwarg artifacts rust is a module-level switch (litellm.rust) that nothing reads from a call's kwargs, so it joins self, use_client and model_config as a registered artifact instead of a DispatchOptions field. The field constants now import from litellm.types.litellm_params directly instead of through a re-export in litellm.types.utils. metadata and litellm_metadata are MutableMapping because their readers mutate them in place, and client accepts raw httpx clients * refactor(types): own max_agentic_loops as an option and walk only nested leaves Move max_agentic_loops from AgenticLoopState to a new AgenticLoopOptions leaf under LiteLLMOptions, since the interception handlers read it as a deployment ceiling rather than stamping it. Drop the owned_wire_names fallback that treated an unresolved annotation as a direct field, which under postponed annotations silently shrank the registry. Re-export TRUSTED_CALLBACK_VARS_FIELD and ADDRESSED_RESPONSE_ID_FIELD from types.utils so that import path keeps working. Tests use hand-written inventories for the callback and pricing names * refactor(types): move data_residency to call state and drop aliased re-exports data_residency is stamped by get_litellm_params and responses.main during the call, so it lives on CallState, not CostOptions. mock_response also accepts a float sequence, which main.py reads for mock embeddings. The types/utils.py re-exports become one plain import with an exact F401 suppression instead of two X as X aliases that pushed PLC0414 over its strict-gate ceiling. Redundant leaf docstrings and the structural artifact test are gone; the re-exported FIELD constants are checked by identity instead * refactor(types): project owned kwarg names once and keep pass-through extraction in request order * refactor(types): type caching_groups from its cache reader and hoist the pass-through ownership set caching_groups is a sequence of flat model-group sequences, which is what Cache._get_caching_group iterates. A regression test drives the public cache key path so two groups in one caching group share a key and a third does not. The pass-through endpoint builds its frozenset of owned names once at import instead of per request, reads the two metadata carriers from the extracted mapping instead of popping them, and its extraction mappings are read-only. Concatenation tests assert the whole derived list and tuple, docstrings drop reader claims that nothing in the module backs * refactor(types): read owned names live in pass-through and pin tests to literal inventories The pass-through endpoint checks body keys against the public all_litellm_params list at request time again, as the base does, instead of a frozenset taken at import, so a name registered after import is still extracted. A test drives that path with a name added after import, and another sends both metadata carriers interleaved with provider keys and asserts the whole merged result. retry_policy accepts the mapping form its router reader builds a RetryPolicy from. The pricing inventory in the typed tests is a literal tuple checked against the model's fields, the agentic compatibility test asserts type, length and set instead of declaration order, and the typed-model overlap tests assert the exact intersection. * refactor(types): move model_alias_map to CallState and read the owned registry in registry order in pass-through * refactor(types): drop restating docstrings, keep FIELD importers on types.utils, pin pass-through registry order * fix(types): satisfy strict lint for public FIELD re-exports * fix(types): restore clean parameter re-exports * fix(tests): compare pass-through extraction order to registry body keys * refactor(types): type owned request parameter leaves * refactor(types): share routing strategy literal and tighten leaf tests * fix(proxy): drop client-supplied proxy-stamped names from pass-through litellm_params * refactor(proxy): name pass-through litellm key split for what it holds * refactor(types): drop TODO markers on the kept readerless fields * fix(types): keep deployment tag_regex and max_file_size_mb out of provider requests * fix(types): include every routing strategy the router accepts --------- Co-authored-by: shrey kharbanda <shreshth@berri.ai> |
||
|
devin-ai-integration[bot]
|
d0d3b6a67e
|
fix(vertex_ai): stop advertising OpenAI platform-only params on Gemma and Llama routes (#43079)
* fix(vertex_ai): stop advertising OpenAI platform-only params on Gemma and Llama routes The Anthropic /v1/messages bridge derives prompt_cache_key from Claude Code's session id whenever the provider config advertises it, and every Vertex OpenAI-compatible route (gemma/, openai/<endpoint>, meta/) inherited the full OpenAI list, so the Model Garden vLLM container rejected each turn with a pydantic extra_forbidden 400. Vertex's Llama and Gemma configs now filter one shared list of platform-only params (prompt_cache_key, prompt_cache_retention, safety_identifier, service_tier, store, web_search_options, modalities, prediction, audio, max_retries) out of their supported params, so the bridge no longer derives the key and drop_params drops an explicit one. * fix(vertex_ai): scope the platform-param filter to self-deployed Model Garden endpoints --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5a75f09d6d
|
fix(vertex_ai): surface the Gemma container's own error inside a 200 :predict response (#43075)
* fix(vertex_ai): surface the Gemma container's own error inside a 200 :predict response * refactor(vertex_ai): move the gemma container error parser next to its adapter --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
25fb7810c2
|
fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token (#40541)
* fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token A CLI session token is a fresh random secret on every login, so since v1.99 each login's spend rows carried a different sha256 hash as api_key and the usage APIs could resolve neither key_alias nor user_email for them. Spend rows and logging callbacks now attribute a session request to its stable alias, cli-session-<user_id>, and the usage endpoints derive that alias and owner from the key itself instead of scanning for a matching digest * fix(spend): resolve the CLI session team from the user's first team in usage metadata A cli-session key carries no team of its own in the DB, so the usage breakdown showed team_id None for it and the export grouped it as Unassigned. The login attaches the user's first team to the session, so the recovery mirrors that rule for cli-session keys only. * fix(spend): claim the session team only for a single-team user The CLI login attaches a team on its own only when the user has exactly one; a user in several teams picks one per login, so usage metadata for the alias would otherwise name a team the login may not have used. * test(pass_through): mark the mocked auth object as a plain key The logged key follows the alias only for a session token; a bare MagicMock reads as one, so the test names the field it relies on. * fix(spend): attribute CLI session pass-through, queue, and managed batch spend to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): only treat the exact cli-session-<created_by> value as a batch key alias A managed object row written by an older build can still carry the raw per-login session token, which shares the cli-session- prefix. Matching on the prefix alone would have surfaced that token as a trusted alias and persisted it verbatim in the batch cost spend log, so the alias check now requires the exact per-user value and every other prefixed value keeps going through redaction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): log proxy executed batch rows under the cli-session alias instead of the session token _row_metadata set user_api_key from the raw bearer token while user_api_key_hash carried the alias, so the spend log redaction rejected the alias as untrusted and hashed the random session token instead Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): attribute semantic search embedding spend to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): scope /key/spend/report for a CLI session to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): use the cli-session alias for websearch spend, prometheus failure labels and the parallel limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(spend): drop explanatory docstrings on get_logged_api_key and attach_user_details Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): only recover cli-session usage keys whose suffix is a known user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e2302be068
|
refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081)
* refactor(ocr): remove the Python OCR execution path and require the Rust route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fmt * refactor(ocr): tidy the native OCR passthrough binding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ocr): ruff format the azure passthrough transformation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ocr): resolve passthrough OCR costing in one Rust call Replace passthrough_url/passthrough_transform with passthrough_response, which matches the relayed endpoint against each Azure config's path segments instead of building a fake request to call get_complete_url. The binding drops the unused headers, status and api_base arguments. Catch the ValueError/RuntimeError the binding raises so a relayed body that is not OCR-shaped falls back to the passthrough object instead of failing logging, and cover the relay against the real binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ocr): drop the unused LlmProviders import from health check helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: drop the ocr_testing job now that tests/ocr_tests is gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ocr): restore the live OCR matrix and the ocr_testing job The public litellm.ocr / aocr / Router interface is unchanged by the Rust migration, so the live provider matrix still applies. Drops the stale VCR skip list for the deleted test_rust_bridge.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ocr): import Final in the health check helper tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
a76f23ac4f
|
fix(bedrock): map Anthropic batch row params the way real time does (#43087)
* fix(bedrock): map Anthropic batch row params the way real time does * fix(bedrock): let a batch row's allowed_openai_params reach the mapper * test(bedrock): assert the batch thinking value matches the real-time mapping * fix(bedrock): keep json_mode out of Anthropic batch rows and pin route-prefixed deployments --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e450f2d54c
|
fix(router): serve Responses turns from a sibling when the encrypted content origin has no boundary peer (#43015)
* test(integration): reproduce encrypted_content_affinity 503 when origin has no boundary peer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep encrypted content affinity turn one out of the response cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): degrade encrypted_content_affinity when the origin has no encryption-boundary peer A routed-group candidate that is currently unavailable and shares its (api_base, api_key) with no healthy deployment used to raise a proxy-level 503/429 from _unavailable_origin_error, even though healthy siblings in the same model group could still serve the turn. Strip the encrypted reasoning and dispatch to the healthy pool instead, matching the existing cross-group behavior, and log a warning naming the origin model_id and routed group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): bound the degraded-affinity log marker and assert the strip on the wire Address review findings: restore num_retries alongside optional_pre_call_checks in the integration test teardown, record scenario request bodies on the scripted upstream so the tests can assert no encrypted reasoning reaches the sibling, and truncate the client-supplied model_id in the degraded-dispatch warning Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): only record JSON bodies on the scripted upstream Multipart uploads to scripted POST routes have no JSON body, so gate the observation recording on the request content-type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): make encrypted_content_affinity runtime-toggleable so /config/update can turn it off --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3fa688223d
|
fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge (#43042)
* fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge Vertex batch uploads treated every non-embeddings JSONL row as a chat completions body, so a /v1/responses row lost its input and reached GCS as a blank text part. Route detection now recognizes /v1/responses rows and bridges them to chat through the same Responses-to-Chat bridge the real-time path uses. That bridge call moves out of the Bedrock files transformation into a shared helper both providers call, forwarding the record's fields as sent, like real time, instead of validating them against the SDK TypedDicts whose required keys clients omit. * chore(batches): type the Vertex responses test helper and drop the quoted input cast * fix(batches): translate developer messages to system on Vertex and Bedrock batch rows like real time --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ba77646991
|
ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests (#42904)
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests * ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * build: point the local proxy unit targets at the nested tests/unit/proxy tree * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
248f0eb159
|
ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903)
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * build: point the local proxy unit targets at the nested tests/unit/proxy tree * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
bf0187072b
|
ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests (#42902)
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
1edc4ba580
|
fix(logging): pass provider response headers to callbacks on every endpoint (#42824)
* fix(logging): pass provider response headers to callbacks on every endpoint Custom callbacks only received kwargs["response_headers"] for chat completions. Responses, image generation and edit, speech, and transcription calls either never recorded the provider's headers or recorded them in one place and not the other. Every handler now records the provider's httpx headers on the response's hidden params as "headers" (raw) and "additional_headers" (processed, with LiteLLM's own entries winning on a clash), and the logging object derives model_call_details["response_headers"] from those hidden params before cost calculation on the non-stream and both streaming success paths, keeping a handler-set value authoritative. Binary speech responses expose their hidden params to the standard logging payload, and the sync OpenAI transcription request always fetches the raw response. * test(images): point the legacy image and speech fakes at the raw response surface Image generation now goes through the SDK's raw response so the provider headers can be read, and the speech binary response now carries hidden params. The unit fakes in the image generation, xinference, proxy provider, image edit, Vertex speech, and otel suites still pinned the old call surface and the old "no hidden params" assertion, so they read an uncalled mock or a fake response without headers. * test(images): drop the rewritten mock comments and the generated edit PNGs * test(images): move the llm-span test's image fake to the raw response surface --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c9e8a04139
|
feat(vertex): native batch JSONL passthrough with cost tracking (#42810)
* feat(vertex): native batch JSONL passthrough with cost tracking Add a per-request `passthrough=true` multipart field on `POST /v1/files` (and the same kwarg on `litellm.create_file`) that uploads a native Vertex AI batch JSONL to the deployment's GCS bucket unchanged, so rows using `googleSearch` and other Gemini-only features run as written and the output, `groundingMetadata` included, comes back untouched. Passthrough is sticky through the GCS object path (`litellm-vertex-files/passthrough/...`), so batch create and output retrieval inherit it without new state. Native output rows are costed from their `usageMetadata` with the deployment's model and model_info, in the polling and retrieve paths and for the existing global `disable_vertex_batch_output_transformation` flag, which billed $0 before. The proxy requires the target to resolve to vertex_ai deployments only, refuses `passthrough` with a non-batch purpose, a non-default `target_storage`, or pre-call guardrails, and validates native rows on `request` instead of the OpenAI batch keys. * refactor(vertex): keep native batch row pricing inside the Vertex adapter Moves native Vertex batch row detection, response parsing, and per-row pricing from litellm/batches/batch_utils.py into litellm/llms/vertex_ai/batches/transformation.py, so batch_utils only aggregates the rows it gets back. Adds tests/test_litellm/files to the misc unit shard so the new test directory is claimed by a shard. * fix(files): say what a passthrough batch upload takes when a row is not native The missing-key 400 listed bare key names, so an OpenAI-shaped row under passthrough=true read "Each line must be a JSON object with keys request". The batch line shape now carries its own hint, and the passthrough one says a passthrough upload takes native Vertex batch rows with a request key * fix(batches): bill native Vertex embedding batch rows on the native cost path A native Vertex output row whose response holds an embedding was validated as a generateContent response, so the documented tokenCount-only shape counted as a failed row. Price embedding rows from their own usage (promptTokenCount, else tokenCount) with the helper the transformed embeddings path already used, and drop the prompt-details helper nothing calls anymore. * fix(batches): keep modality batch rates on native Vertex embedding rows An embedding row that carries usageMetadata was billed from promptTokenCount alone, so its promptTokensDetails no longer reached the audio, image, and video batch rates the way it did before the native cost path. Run every row with usageMetadata through the Gemini usage parser and keep the flat tokenCount fallback for embedding rows without it. * fix(batches): price native Vertex batch rows by modelVersion under a wildcard deployment A `vertex_ai/*` deployment hands the batch cost path `*` as the deployment model, which no cost map resolves, so every native (passthrough or flag-on) row was billed at $0. A wildcard deployment model now defers to the row's own `modelVersion`, the way the transformed path already prices by the row's `model`. Also moves the native passthrough tests under tests/test_litellm, the tree codecov reads, and covers the raw upload chunking, the embedding output translation, the unpriceable-row path, and the flag-on dispatch. * fix(batches): keep explicit deployment prices for native Vertex rows without a modelVersion Under a wildcard deployment a native batch row that carries no modelVersion (an embedding row, or a generateContent row Vertex returned without one) was billed at $0 even when the deployment's model_info sets explicit batch prices, because the cost calculator was never called. The row now falls back to the wildcard name, which the cost calculator prices from the explicit model_info, and only a row with neither a modelVersion nor a deployment model is billed at $0 with the warning --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
f4f1a75a9c
|
test(vertex_ai): run the files peak-memory guards without coverage tracing (#42914)
The new CircleCI tests pipeline (#42773) runs tests/unit under pytest-cov on CPython 3.12.2, where coverage traces every line through sys.settrace. The two tracemalloc peak comparisons in test_vertex_ai_files_streaming.py drive 8000-row payloads through both pipelines and slow from ~10s to over 3 minutes under that tracer, so both hit the 90s pytest-timeout on every run. Mark them no_cover so pytest-cov pauses tracing for just these two. Their assertions are unchanged and every other test in the file still reports coverage. |
||
|
devin-ai-integration[bot]
|
3fb6f8740b
|
test(integration): add read-replica routing harness to the CircleCI integration suite (#42692)
* test(integration): add read-replica routing harness * refactor(integration): hoist the maintenance url imports * fix(integration): keep per-test databases and the witness sequence readable under replica roles * fix(integration): opt bespoke database and pool tests out of the injected read replica * test(integration): commit recorded replica routing expectations * fix(integration): judge routing by role containment so shrinking role sets do not fail * fix(integration): run the pool-limit shutdown choreography on the superuser database url * ci(integration): add the mcp group to the replica matrix * fix(integration): judge routing by exact role sets with a named either-role allowlist * test(integration): drop containment-era routing expectations for re-recording * chore(integration): drop docstrings from the replica harness scripts * docs(integration): describe exact routing matching and the either-role list * test(integration): record exact replica routing expectations * test(integration): allow the SELECT 1 health probe on either role * test(integration): replace committed routing expectations with an on-demand base-vs-head parity run * test(integration): fix parity env scope, readme wording, and seed-deterministic serialization test * test(integration): make the sorted-role serialization test deterministic in-process * test(integration): swap all product code in parity runs and pin role gains * ci(integration): force tracked-file removal before parity checkout --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
d248cc5914
|
fix(fireworks_ai): route firerouter short names and bill pass-through legs at the routed model's rates (#42814)
* fix(fireworks_ai): route firerouter short names and bill pass-through legs at the routed model's rates fireworks_ai/firerouter and fireworks_ai/firerouter/<slug> resolve to accounts/fireworks/routers/... instead of a models/ path, and the cost calculator falls back to the routed model's own catalog entry before the Fireworks size buckets so a Claude leg is no longer priced at $0 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(fireworks_ai): bill routed legs under the routed model's own provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fireworks_ai): require the k suffix when parsing tiered input fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1175559c39
|
feat(lint): add LIT013 flagging *-ok suppressions that suppress nothing and remove the 240 stale ones (#42793) | ||
|
devin-ai-integration[bot]
|
c2b388ebe6
|
fix(bedrock): honour stream_chunk_size in Invoke streaming (#42686) | ||
|
devin-ai-integration[bot]
|
e73f949fbb
|
fix(params): stop stream_chunk_size reaching provider request bodies (#42664)
* fix(params): carry stream_chunk_size through litellm_params instead of provider params * test(integration): fence stream_chunk_size out of every provider request body * test(bedrock): type parametrized stream chunk test params * test(integration): drop the contracts manifest resurrected by the main merge * test(bedrock): type the stream_chunk_size test helpers * test(params): finish AGENTS.md typing pass on stream_chunk_size tests * test(integration): drop the covers marker from the stream_chunk_size wire test --------- Co-authored-by: shrey kharbanda <shreshth@berri.ai> |
||
|
devin-ai-integration[bot]
|
5c24802fbd
|
fix(bedrock): send json_schema as a forced tool on Claude Opus 4.7 and 4.8 Converse (#42644)
* fix(bedrock): send json_schema as a forced tool on Claude Opus 4.7 and 4.8 Converse Bedrock rejects outputConfig.textFormat on Opus 4.7 and 4.8 with "output_config.format: Extra inputs are not permitted", and the AWS model cards list structured outputs as not supported for both, so their cost-map entries no longer claim supports_native_structured_output and json_schema requests fall back to the json_tool_call tool. Fixes #27846 * test(bedrock): assert Opus 4.7 and 4.8 inline the schema on Invoke, move the native case to Sonnet 4.6 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
4f93e2c3da
|
test: point CircleCI-only suites at models still in the cost map (#42617)
* test: point CircleCI-only suites at models still in the cost map #42435 removed cost map entries past their deprecation date and #42437 added litellm_uisettings to the config-synced tables, but both only updated tests/test_litellm. The CircleCI-only suites (local_testing, llm_translation, logging_callback_tests, litellm_utils_tests, unit) kept using the removed models or the old table list and went red on main. Each test keeps its assertions and swaps the removed model for a current one with the same provider and capabilities. The fireworks tests pick a vision model from the cost map because #34941 set supports_vision false on minimax-m3, and the vertex image provider test injects the image model set because #42435 removed every vertex_ai-image-models entry. * test(vertex_ai): register the image model through add_known_models in the provider test |
||
|
devin-ai-integration[bot]
|
075536eca1
|
chore(cost-map): remove models past their deprecation date (#42435)
* chore(cost-map): remove models past their deprecation date Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost-calc): drop the empty parametrize left behind by the gemini web search removal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): drop merge base block left by conflict resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost-calc): drop gemini image cost tests pinned on removed model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4b65ef6d64
|
test(response_metadata): make the detailed-timing receive-anchor test timezone independent (#42429)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
deba473821
|
fix(cost): bill batch prompts above 272K at OpenAI's long-context batch tier (#39861)
* fix(cost): bill batch prompts above 272K at OpenAI's long-context batch tier * fix(cost): mirror batch long-context keys on custom pricing params Register the two *_above_272k_tokens_batches keys on CustomPricingLiteLLMParams so a per-deployment override stays out of the shared backend key, add them to the inline model-info schema and alias-count tests, and build LiteLLM_Params and GenericLiteLLMParams through model_validate at the two dict-splat call sites so basedpyright's reportArgumentType budget ratchets down instead of blocking the new fields. * fix(cost): add the gpt-5.5-pro batch long-context tier and ignore malformed batch tier keys * fix(cost): bill cached batch tokens at OpenAI's cached batch rate Adds cache_read_input_token_cost_batches and cache_read_input_token_cost_above_272k_tokens_batches for the tiered OpenAI entries at half the standard cached rate, bills cached batch tokens at that rate per output line, and parses string-valued batch rates in deployment-level model_info. * fix(cost): bill batch cache writes at the batch cache-write rate and carry published batch rates for one-sided deployments OpenAI's Batch table prices cache writes for gpt-6-astra, gpt-5.6, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna at half the standard cache-write rate, so the cost map gains cache_creation_input_token_cost_batches and its above_272k tier for those entries and batch cost pulls written tokens out of the input bucket at that rate; models without the key keep billing writes at the batch input rate. A deployment declaring only one side of its batch pricing now carries every published batch rate of the other side (tier, cached, cache write), its own keys win, and a lone tier, cached or cache-write batch key counts as declared pricing instead of being ignored. * fix(cost): select the batch long-context tier from any batch tier key A deployment that declares its own flat standard input rate keeps every published batch rate of the output direction, including the 272K output tier, but the tier was only ever selected when an input tier key was also present. Detect the crossed tier from any of the four batch tier keys so the carried output, cache-read, and cache-write tiers bill at their tier rate above 272K tokens. * chore(proxy): keep the OpenAPI snapshot as CI generates it * fix(cost): pick each batch price component's tier from its own keys The batch rate picker crossed one threshold for every component, so a deployment declaring only an output tier also moved its input, cached, and cache-write rates to that cutoff. Each component now crosses its own *_above_<N>k_tokens_batches keys and falls back to its flat key. The JSON schema is regenerated with the generator as it is on main: cost-map-guard renders the PR's cost map with the base branch's generator, so the descriptions for the new batch cache keys move to a follow-up. * chore(proxy): restore the lazy OpenAPI snapshot to what CI's Python 3.12 generates The merge commit carried a snapshot regenerated on a Python 3.14 venv, which dedents docstrings at compile time, so one description line differed from the file CI regenerates on 3.12 and the schema.d.ts sync check went red. The snapshot is byte-identical to main again |
||
|
devin-ai-integration[bot]
|
691c0d6596
|
test(unit): make bedrock collector and secret scan timing tests deterministic (#42405)
* test(unit): make bedrock collector and secret scan timing tests deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): count interpreter calls instead of wall clock in the secret scan scaling test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): profile the secret scan with cProfile, restore the outer profiler and tighten the scaling bound Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8603d6e259
|
fix(cost): honor per-second custom pricing on chat completions for every provider (#42403)
* fix(cost): honor per-second custom pricing on chat completions for every provider * test(cost): assert a per-second priced deployment bills instead of staying at $0 The zero-cost diagnostic tests from #42345 used a per-second-only entry as their misconfigured fixture, which this branch now bills. Switch that fixture to a per-query-only entry, which is still selected as the deployment's own pricing and still prices chat usage at $0, and add a per-second test asserting the call duration is billed with no diagnostic Also let a caller's explicit total_time outrank the logging window in completion_cost, so the SDK precedence stays stamped response, caller, logging * test(response_metadata): move the per-second pricing regression into the mapped tests/unit file * fix(cost_calculator): keep media-mode per-second rates off the wall-clock path A video, transcription, speech, or realtime entry's per-second rates price media seconds, which their dedicated cost paths bill from the media itself. The generic per-second branch now skips those modes, so a video status poll on a per-second video model bills nothing instead of the seconds the poll took to answer. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3106d9c573
|
feat(rust-bridge): add cache and secret migration foundations (#42328)
* docs(rust): plan Python interop foundation * fix(rust): preserve Python settings coercion at the native boundary * chore(rust): drop interop planning note Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(rust): resolve OCR provider secrets through an async SecretSource before transformation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(rust): project the Python secret manager into the bridge and resolve OCR secrets through it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(rust): drop premium_user from the secret manager snapshot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rust-bridge): read the private key management globals once in the settings snapshot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rust): bound the bridge secret manager state cache to the active snapshot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust): inline coercion unit tests * fix(rust): preserve Python secret manager bindings * refactor(rust-bridge): let settings projectors own their contract specs Each settings group now declares its SettingSpec rows next to the projector that reads them, and the manifest test derives python_settings.json from those tables instead of a hand-copied duplicate. Field carries (group, name) instead of a dotted path, and coercion gains the dict-item reader plus the Redis Boolean, certificate-requirement, non-empty string, and numeric adapters that the cache configuration projection adopts next. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(rust-bridge): capture the secret manager binding in one settings read The secret_manager accessor now carries the live client and settings objects, so the bridge classifies the binding from a single snapshot instead of re-reading litellm globals. The unreachable native arm and the service alias go away, the binding-to-state mapping moves next to the snapshot, and the Python callback precomputes its key_manager name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(rust-bridge): execute typed settings field declarations * refactor(rust-bridge): compare cache backends by identity behind one exact trait cache-response gains an object-safe ExactResponseCache so every exact-match backend sits behind one pointer; WriteBuffer flushes through it. The bridge's NativeResponseCache shrinks from nine variants and fifteen per-backend accessors to an exact service plus the three semantic backends, and facade mismatch detection compares BackendIdentity values instead of matching on each backend type. Request projections move next to NativeRequest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(rust-bridge): drive both Python-embedded semantic caches through one execution Redis-semantic and Valkey-semantic operations now share one SemanticExecution body: await the Python embedder, seed the task-local vector, run the native backend, repeat per batch entry. Valkey drops its with_embedder path in favor of the same seeded embedder, and each backend keeps its own embedding-failure policy. PythonEmbedder exposes one call shape. Redis-semantic thresholds are compared at the backend's f32 width, which un-breaks the redis-stack parity tests that a 0.8 facade threshold failed before this branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * wip * feat(rust-bridge): complete response cache runtime surface * fix(rust-bridge): preserve secret manager callback exceptions * refactor(rust-bridge): unify route cache and secret rollout catalog --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
5dc6261ebb
|
fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key (#42342)
* fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): keep S3 signer test additions scoped to new cases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): drop e2e suite changes from the S3 signing fix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): build S3 credentials directly from the s3_* pair so ambient AWS_* env never mixes in Restores the split-identity e2e coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan
|
baee50546f |
chore: merge main into litellm_cherry_pick_password_breach_reset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
e8d97d381a
|
Merge pull request #42080 from BerriAI/litellm_prompt_caching_affinity_lookback
fix(router): keep prompt caching affinity when the breakpoint moves |
||
|
Mateo Wang
|
0e7cf5113e
|
Merge pull request #42069 from BerriAI/litellm_redacted_thinking_prompt_caching_pin
fix(token_counter): count replayed redacted_thinking blocks so prompt_caching keeps pinning |
||
|
Mateo Wang
|
9a90adad32
|
Merge pull request #42275 from BerriAI/litellm_claude_platform_messages_beta_passthrough
fix(bedrock): forward anthropic-beta headers verbatim on the Claude platform messages path |
||
|
mateo-berri
|
5ea4fe620f | test(router): give each prompt caching check test a fresh callback registry | ||
|
mateo-berri
|
f549c87091 | Merge branch 'main' into litellm_prompt_caching_affinity_lookback | ||
|
ryan
|
3fe405a5cc |
feat(auth): breached password detection, self-service change-password and forced password reset
Cherry-pick of merge commit
|
||
|
mateo-berri
|
e51ccbc759 | fix(bedrock): forward anthropic-beta headers verbatim on the Claude platform messages path | ||
|
Moe Khalil
|
83ec5d6101 |
fix(auto-router): skip JEV for encrypted delegated tasks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
79d1af9d3c | chore: merge origin/main to pick up the pre-call check test move | ||
|
Moe Khalil
|
bb46e8b774 |
chore(auto-router): merge main and preserve JEV configuration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
58729ac69f |
test(a2a): sort imports in merged bedrock agentcore test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |