mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
155 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
b2e82cf3be
|
fix(caching): stand default cache points down when extra_body hides a direct client mark (#43341)
* fix(caching): stand default cache points down when extra_body hides a direct client mark On native /v1/messages the extra_body envelope is dropped, so a client tool mark or root cache_control reaches Anthropic even when extra_body overrides it. The stand-down check only counted the envelope-merged view and injected two default marks on top of the client's. * fix(caching): keep chat completions on the envelope-merged mark count for the default stand-down Chat completions merge extra_body over the request, so a direct tool mark that extra_body replaces never reaches the provider there. Only /v1/messages, where the native transforms drop the envelope, needs to count marks on both sides. |
||
|
devin-ai-integration[bot]
|
849f3037b4
|
fix(langtrace): deliver spans to app.langtrace.ai/api/trace with x-api-key (#43322)
* test(langtrace): integration test for the built-in callback wire (path, x-api-key) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langtrace): deliver spans to app.langtrace.ai/api/trace with x-api-key The built-in langtrace callback posted to the dead host langtrace.ai, sent the key as api_key instead of x-api-key, and let the OTLP endpoint normalizer append /v1/traces to the complete /api/trace path, so every export returned 404. Default the host to https://app.langtrace.ai, honor LANGTRACE_API_HOST for self-hosted servers, pass the key as an exporter header instead of a process-wide env var, and keep the /api/trace path unchanged for traces on the langtrace callback only Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langtrace): keep LANGTRACE_API_HOST that already ends in /api/trace Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langtrace): deterministic audit inventory for the built-in callback (surfaces, failures, endpoints, chaos) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langtrace): build the repeated-request body once Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langtrace): outage cell asserts at-most-once delivery, not span loss The OTLP HTTP exporter reposts once on ConnectionError and the batch processor may still be flushing the previous burst when the sink closes, so whether the outage burst is lost or delivered after revival depends on timing. The invariant is no duplicate and recovery on the same port Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langtrace): delivered spans must carry the prompt in the gen_ai.content.prompt event The upstream echoes the marker into the completion, so a whole-span match alone would still pass if the prompt event disappeared Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langtrace): disable model info refresh so the scripted upstream only sees completion requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b248b1c7dc
|
fix(openai): exclude fine-tuned and custom gpt-5-chat aliases from gpt-5 reasoning path (#43185)
* fix(openai): exclude fine-tuned and custom gpt-5-chat aliases from gpt-5 reasoning path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai): keep gpt-5-chat alias regression test diff minimal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai): cover temperature pass-through for gpt-5-chat aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai): annotate locals and wrap long lines in gpt-5-chat alias test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
shrey-berri
|
e494723105
|
fix(params): validate stream_chunk_size once, before any provider call (#43222)
* fix(params): validate stream_chunk_size once and carry it as typed control options Checks stream_chunk_size at the top of completion() and acompletion(), accepts digit strings, returns a 400 naming the param unless drop_params is set, and stores the checked value under _litellm_control. Bedrock Converse and Invoke read it from litellm_params; the Bedrock-only checker and the dead Invoke pops are gone. Owned-kwarg filtering now runs through one helper everywhere. Refs LIT-8317 * test(bedrock): drop tests for the removed stream_chunk_size_from helper Refs LIT-8317 * fix(params): check stream_chunk_size before the MCP gateway branch Refs LIT-8317 * fix(params): return assert_never in the exhaustive control-options match Refs LIT-8317 * fix(params): address council review of the control options change Read all_litellm_params live so names registered after import stay LiteLLM-owned, make litellm_params a required keyword on the stream wrapper hooks, give digit strings and ints the same 18-digit range, share the default-chunking test table, test the Responses bridge through litellm.responses, and revert formatting-only churn in existing tests. Refs LIT-8317 * fix(params): address the second council review of control options Keep the Responses bridge on its original all_litellm_params forwarding, narrow _int_from_decimal_string inline so it type-checks, bound nested huge ints in the error message, store _litellm_control only when a value is set, simplify the parser to its single field, drop the one-caller wrapper, and tighten the tests. Refs LIT-8317 * fix(params): keep the 18-digit length check on stream_chunk_size strings A 19-character string with leading zeros such as 0000000000000000001 would otherwise pass as 1, although the rule and the error message say at most 18 digits. Refs LIT-8317 * test(params): tidy control options tests after council sign-off Move the Responses bridge test into the existing bridge test file, drop the rebind test that pinned an implementation detail, assert through stored_control_options instead of the storage key, and cover drop_params="true" through Bedrock streaming. Refs LIT-8317 * test(params): wrap a chunking test row that went past 120 characters Refs LIT-8317 |
||
|
devin-ai-integration[bot]
|
8d166258a6
|
fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346)
* fix(tests): stop VCR recording and replaying a test's own localhost upstream * test(vcr): prove a localhost response an earlier run stored is never replayed * test(vcr): drive the localhost cassette checks in-process instead of through a loopback server --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
635a718ba1
|
ci: cut CircleCI wall time without loosening test isolation (#43347)
* ci: cut CircleCI wall time without loosening test isolation * fix(ci): parse integration split files that follow --results The CircleCI machine image ships Python 3.12.2, whose argparse leaves the files positional empty when it follows an option and another positional, so every extensions node exited with 'unrecognized arguments'. Reproduced on 3.12.2; parse_intermixed_args selects the files on 3.12.2, 3.12.13 and 3.13 * test(ci): resolve command references in the Rust toolchain guard The Windows rustup install moved into the install_windows_toolchain command, which the guard only recognized for install_rust. It now accepts any command that installs a pinned rustup and reads the Windows toolchain pin from it * ci: cache the Windows release cargo build from main windows_release_wheel rebuilt every dependency with fat LTO on each run. It now restores the release target and cargo registry saved by main's scheduled run, drops the workspace crates' fingerprints so they always rebuild from the checked-out source, and still runs the full LTO link * ci: run the Windows release wheel build on windows.xlarge The fat-LTO release build is the slowest job in the pipeline; more cores speed up the dependency compile ahead of the final link * ci: skip the Windows fingerprint cleanup when the cargo cache missed On a cold cache the release fingerprint directory does not exist, and the CircleCI PowerShell wrapper failed the step on the suppressed not-found error |
||
|
shrey-berri
|
9413b82477
|
fix(params): keep _litellm_* kwargs out of provider request bodies by construction (#43221)
Kwargs LiteLLM code introduces for its own use were only kept out of provider bodies if someone also listed them in all_litellm_params. Undeclared ones went into extra_body or optional_params, reached the provider, and the provider rejected the request. is_litellm_owned_kwarg in types/utils.py now defines LiteLLM-owned once: a registered name, or any name starting with INTERNAL_KWARG_PREFIX from litellm/constants.py. Every filter that builds provider params from kwargs uses it: chat completion, transcription, embedding, image generation and edit, search and video, ElevenLabs text to speech, and the Bedrock batch mapper. The two untyped shared filters now take Mapping[str, object] The stream_chunk_size wire test becomes test_internal_params_wire.py. It also sends an undeclared _litellm_ kwarg and asserts that no _litellm_ key reaches any of the six provider bodies, while extra_body passthrough keeps working Refs LIT-8318, LIT-8319 |
||
|
devin-ai-integration[bot]
|
e47b1f2a3f
|
fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022)
* fix(s3_v2): drop terminal upload failures, bound retries per flush and enforce the queue cap at enqueue Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): keep retrying credential-rotation 403s, only AccessDenied-style errors are terminal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(env_keys): exclude DEFAULT_S3_MAX_FLUSH_ATTEMPTS as an internal tuning var Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): retry every 5xx, warn on first queue overflow, validate the flush budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): read the queue cap defensively so un-initialized loggers still enqueue Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): drop the getattr in _enqueue and tighten the retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): keep the constructor flush budget when the callback override is invalid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(s3_v2): adapt per-object upload concurrency to sink latency and throttling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(s3_v2): tidy adaptive limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(s3_v2): wake one waiter per released upload slot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(s3_v2): make the enqueue queue cap configurable with s3_max_queue_size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): audit cells for cache hits, coded 403 and callback modes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): retry bucket-wide failures by default, age-budget requeues and make terminal drops and adaptive concurrency opt-in Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(s3_v2): suppress the missing-waiter ValueError explicitly in the adaptive limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): count oldest events trimmed after a failed flush as callback failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(s3_v2): move the mutable-ok marker onto the list literal it suppresses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): report post-flush overflow drops once and grow adaptive concurrency above the floor before asserting back-off Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): fail the SlowDown back-off test when the measured window sees no PUTs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): restore base retry defaults, opt-in age budget, no enqueue cap, back off outside the limiter slot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): hoist the default no-op upload slot to a module constant Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): drop unused mutable-ok suppressions on queue appends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): keep the retry queue oldest-first and prioritise fresh events at upload time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(s3_v2): keep the mutable-ok marker on the queue list literal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): build request bodies inside the upload slot and keep the sync retry set at base parity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): drop wall-clock sleeps from the unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): rebuild the request body inside the slot on every retry attempt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): anchor the backoff window on the first observed failure and tighten shard assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): default the upload slot to the logger limiter so monkeypatched doubles keep working Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): clear ambient AWS env credentials so the rotating profile signs the sync retry test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): shrink the linear send-batch perf test to 2k/8k elements Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): fix stale batch sizes in the perf test assert message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): keep async in-call retries on the base 403/500/503 set Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): hold the upload slot across retries, restore the bool upload contract, and fail safe on bool config Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): mark dropped uploads by element identity so a shared key cannot mask a retryable sibling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): make the per-flush drop lookup constant time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): take the upload slot in the caller like base, build the body once per attempt loop Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): match base retry, logging and hook behaviour unless the new options are opted in Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): assert the wire key in the init-bypassed sync upload test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): assert the signed headers and wire key in the init-bypassed sync upload test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): default the upload limiter at class level instead of reading it with getattr Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): drop the duplicate annotations that redeclare the class-level counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(s3_v2): drop terminal-failed uploads by default and bound retry age to one hour Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): fall back to the configured retry age on invalid values Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): drive retry-age tests from a fixed clock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): audit cells for retry-age opt-out and 429 single-put parity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
26bf575f15
|
feat(guardrails): scan retrieved vector store chunks with the request's pre-call guardrails (#43271)
* feat(guardrails): scan retrieved vector store chunks with the request's pre-call guardrails Vector store retrieval runs inside acompletion after the proxy's pre-call guardrails have already seen the request, so a retrieved chunk carrying an injection reached the prompt unscanned. Each retrieved context message now goes through every pre-call guardrail the request is subject to before it is injected: a block raises the same 400 the guardrail gives for user text, a masking guardrail rewrites the context, and a guardrail that fails while scanning fails the request instead of injecting the chunk unscanned * fix(guardrails): return a guardrail block unmapped from exception_type so the Responses API surfaces the guardrail's own 400 * fix(guardrails): build the deployment hooks' identity from stamped metadata only Top-level user_api_key_* fields in a request body are client controlled, so the pre-call, chunk scan, and post-call deployment hooks now take UserAPIKeyAuth from the metadata the proxy stamped, and the chunk scanner returns or raises on every branch. * fix(guardrails): block route verdicts on retrieved chunks, keep guardrail verdicts out of router retries and fallbacks, and scan chunks against the client's request * fix(guardrails): keep the merged guardrail list when scanning chunks against the client's request The scan request laid the client's kept body over the deployment kwargs, so a client that sent its own top-level guardrails list shadowed the merged metadata.guardrails list and a key or team guardrail skipped the chunk scan. The kwargs now win and the keys the proxy relocates into metadata are dropped from the body's contribution. * fix(guardrails): strip the deployment's guardrail keys from the chunk scan request so merged team guardrails still run * test(guardrails): type the vector store scan test doubles * test(guardrails): type the scan double's request data --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
96c008f420
|
ci: fail on new unbounded SQL IN lists and add a Prisma chunking helper (#42629)
* ci: warn on SQL IN lists with no written bound Postgres caps a prepared statement at 32,767 bind parameters and a membership filter binds one per value, so an IN list built from table data breaks once the table outgrows the cap. That is how the budget reset job froze every due budget (LIT-7535, #40564). check_unbounded_in_lists.py reports every Prisma "in" / "not_in" filter whose value has no fixed size and every raw SQL literal that splices a list in after "IN (", unless the line carries "# bounded-ok: <reason>". It only warns for now: the output is the inventory for RCA action item AI-1, and it exits 0. * ci: decide a constant IN list by its module binding, not its casing An ALL_CAPS name imported or filled at runtime is as unbounded as any other, so a name now passes only when the module binds it once to a value of fixed size. Adds Final to the locals a loop does not forbid. * ci: only a frozen module value makes an IN list constant A module list bound once could still grow through append or extend, so a name now counts as fixed only when it is bound to a tuple, frozenset or constant. Trims the module docstring to what a reader needs. * ci: chunk Prisma IN lists with a shared helper and fail on new unbounded ones Add litellm.repositories.bounded_in: find_many_in, count_in, update_many_in and delete_many_in split a deduplicated value list into 5,000-value chunks, AND each chunk with the caller's where, run them in order (a transaction handle works) and combine the results. Writes take a required atomicity argument, and a where that already filters the chunked field is refused. check_unbounded_in_lists.py now fails CI on any finding missing from unbounded_in_baseline.txt and on any stale baseline entry, so the baseline only shrinks. Entries are keyed by path, enclosing scope, kind, field and occurrence, not line numbers. The helper module is exempt, a constant spread into a frozen tuple counts as fixed, and messages point at the helper for "in" and at an array parameter for "not_in" and raw SQL. A real-Postgres integration test shows a raw 40,000-value filter rejected for too many bind variables while the helpers handle it. * refactor: rename bounded_in to chunked_in and let callers pick a chunk size The helper module is litellm.repositories.chunked_in, and its unit and integration tests, the checker's exemption path and its finding messages follow the new name. The `# bounded-ok` marker is unchanged. find_many_in, count_in, update_many_in and delete_many_in take a keyword-only chunk_size, defaulting to IN_LIST_CHUNK_SIZE (5,000). A value below 1 or above MAX_IN_LIST_CHUNK_SIZE (30,000) raises ValueError before any query, which leaves the rest of the filter headroom under Postgres's 32,767 bind-parameter cap. * refactor: flatten chunked_in's stacked comprehensions with chain.from_iterable LIT014 (#42650) caps a comprehension at one for and one if clause. The four nested walks in the helper now chain their iterables instead, with the same order and results. * refactor: recover user details with find_many_in, sending chunks as lists _details_for_user_ids reads users through find_many_in instead of a raw "in" filter, so its lookup stays under the bind-parameter cap for any number of recovered keys. Up to 5,000 ids it still sends one find_many with the same where dict, and a PrismaError from any chunk is still logged and treated as no details. The helper now sends each chunk as a list, so a chunked filter equals the dict a hand-written call would send and a migrated call site's existing assertions keep passing. The site's baseline entry is gone. * ci: skip functional TypedDict field maps in the unbounded IN list check The dict passed as the field map of TypedDict("Name", {...}), or as its fields= keyword, names fields: an "in" or "notIn" key there is a type, not a filter. Only that dict is skipped, for TypedDict, typing.TypedDict and typing_extensions.TypedDict; a filter nested in a field value or passed to any other call is still reported. The two types/proxy/management_endpoints/team_endpoints.py entries leave the baseline, which is now 156. * fix: refuse an update_many_in whose data writes the chunked field Chunks run one after another, so an update that sets the chunked field can move a row into a later chunk, which updates it again and counts it twice: values ["old", "new"] with chunk_size=1 and data={"id": "new"} does exactly that. update_many_in now raises ChunkedFieldWriteError before any query when data has the chunked field as a top-level key, in any form, including Prisma operators such as {"set": ...}. * docs: cut the unbounded IN list checker's docstring to what it flags and how to clear it It now says what is reported, the three ways to clear a finding, and how the baseline and --update-baseline work, in 11 lines. The per-shape detail lives in the tests. * ci: key an unbounded IN list finding by its filtered expression too A baseline key of path, scope, kind, field and occurrence let a PR delete a baselined filter and add a different unbounded one on the same field in the same function, and the new one took over the old key. The key now also carries the filtered expression's source, whitespace-normalized (the Prisma value, or a raw-SQL `IN (...)` slot), so that swap reads as one new and one stale entry and fails the run. The same expression re-added in the same function is still the same finding. Every baseline entry is rewritten in the new form; the 156 findings are unchanged, and only occurrence indexes renumber where one field had several different expressions. |
||
|
devin-ai-integration[bot]
|
69ad004015
|
refactor(anthropic): rename experimental_pass_through to pass_through (#43329)
* refactor(anthropic): rename experimental_pass_through to pass_through Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): point compact patch targets at renamed pass_through path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3a6744cd02
|
feat(sail): add Sail as a provider with service_tier mapped to its completion window (#42840)
Register Sail (providers.json, LlmProviders.SAIL, OpenAI-compatible lists, ProviderConfigManager) for chat, Responses and /v1/messages, and add its 12 models to both cost maps with asap, balanced and flex price columns. Sail picks speed and price with metadata.completion_window and rejects service_tier, so the Sail chat and Responses configs translate the tier: default and priority to asap, flex to flex, balanced to balanced, auto to no window. Billing prices the window that was sent. A tier Sail has no window for, or a window or tier set where billing cannot see it (request metadata, extra_body), is a 400 unless drop_params is set. Add balanced to ServiceTier and its _balanced price columns to the model info types, the Rust catalog and the dashboard schema. A transform_extra_body hook on the chat and Responses base configs, which returns extra_body unchanged by default, lets Sail keep the window when a caller also sends extra_body.metadata. Sail is listed in the Add Model form and model picker. Co-authored-by: shrey kharbanda <shrey@berri.ai> |
||
|
devin-ai-integration[bot]
|
5ac640e49d
|
fix(responses): run stream failure and success hooks on the iterating loop instead of blocking it (#43270)
* fix(responses): run stream failure and success hooks on the iterating loop instead of blocking it A dropped provider stream on native /v1/responses ran the failure logging through run_async_function from inside the async iterator, which parks the event loop thread on a helper-thread future until every failure callback returns, and never returns when a callback waits on state only that loop can advance. With a running loop the failure handlers (and the completed-stream success deployment hook) are now scheduled as tasks on it, the way chat streaming already does; the sync iterator keeps its blocking path * fix(responses): await stream failure and success logging on the iterating loop before propagating Keep the merge-base hook set for the native Responses stream: async_failure_handler plus the executor-thread failure_handler on failure, and the post-call success deployment hook on completion. Inside a running loop the async handler is scheduled as a task on that loop and the async iterator awaits it before re-raising, so the loop is never blocked on a foreign-loop future and the failure is attributed before the router's fallback wrapper re-enters the same logging object. The sync iterator inside a running loop keeps the task fire-and-forget with a strong reference. * fix(responses): submit the sync failure handler only after the async one finishes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
9540f19e38
|
feat(proxy): add fail_closed_rate_limit_enforcement to reject requests with 503 while Redis rate limit counters are unreachable (#43251)
* feat(proxy): add fail_closed_rate_limit_enforcement to reject requests with 503 while Redis rate limit counters are unreachable * fix(proxy): reject fail-closed rate limit checks before logging the in-memory fallback and pin the boot warning in the lifespan * fix(proxy): coerce the fail-closed flag, fail closed on read-only checks, and refund partial cluster increments * fix(proxy): window-guard rate limit refunds and catch the fail-closed rejection by type * fix(proxy): read the compaction rate-limit gate's limiter from the proxy hook registry * fix(proxy): count the pending request in read-only rate-limit checks and keep the compaction gate off the caller's parallel slot The compaction polyfill's summary-model gate, once it ran against the real v3 limiter, showed two behaviors nobody had chosen. The read-only check compared the stored counter with the same `>` the increment path uses, but a read-only check decides a request that has not been counted yet, so a summary model exactly at its rpm limit still went out. The read-only path now adds the pending increment of 1 before comparing; the increment path is unchanged. The gate also passed the key's max_parallel_requests gauge through, and the read-only gauge count includes the caller's own in-flight slot, so a key with max_parallel_requests: 1 never compacted. The gate now drops that gauge from its descriptors, since the summary call runs inside a request the limiter already admitted. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d18fcb09d6
|
fix(otel): detach post-response service spans by request phase, name redis spans by operation (#43237)
* fix(otel): detach post-response service spans by request phase, name redis spans by operation Service spans logged from the post-response phase (success callbacks, the response-cache write) now root their own trace linked to the request span even while the server span is still recording, instead of only when they happen to end after it. Redis service spans are named `redis <operation>`; the litellm call chain that issued them moves to the `litellm.service.caller` attribute via a typed `ServiceLoggerPayload.caller` field. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep the service caller on failure and legacy spans, test the production phase dispatch sites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): mark anthropic messages stream cache write as post-response phase The /v1/messages streaming cache writer awaits async_add_cache inline instead of going through create_cache_write_task, so its redis span stayed parented under the request trace. Wrap the write in post_response_phase so it detaches like the chat completions write. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): write the Messages stream cache in a background task after handoff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
14f4c34c61
|
fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294)
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts The Langfuse redaction unit test exports to a local OTLP capture instead of polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill test only requires spend rows for requests the surviving worker served. The spend-routes sweep treats the intentional /spend/capture_rate 503 as expected. CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper instead of reading it as "variable exists". The integration egress guard now matches the script's own cgroup, so it no longer blocks the CircleCI agent, which runs as the same user. * fix(ci): keep the policy-load backoff typed as float * fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests * fix(secrets): load CyberArk policy one request at a time per manager * test(secrets): pin that non-conflict CyberArk policy failures are not retried * test(unit): run tests/unit with only an allowlisted host environment CircleCI's unit job inherits every project env var, so real provider keys, REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that assume none are set. Locally, litellm's import-time load_dotenv did the same from any .env up the tree. The unit conftest now drops every variable outside a small allowlist and disables dotenv before litellm is imported. * test(e2e): name a failed search and the stuck batch status instead of misattributing them The websearch session test read an empty web_search_tool_result_error block as a successful search, so a failing search tool surfaced as a session billing bug. The batch cancellation timeout now reports the last status the proxy returned. * fix(ci): scrub the host environment per unit test instead of for the whole pytest process GHA shards run tests/unit next to other suites in one process, so the import-time scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP upstream fell back to the SDK2 interpreter. The two websearch tests that called OpenAI and Perplexity live are removed: tests/unit no longer sees their keys. * fix(ci): scrub only the host variables present before litellm is imported The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to its bundled encodings, so tokenizer paths tried to download them and hit the socket guard. The prisma setup test now passes its own database URL instead of reading one another test leaked into the process environment. * fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue LoggingWorker marked a task done on whichever queue was current when the callback finished, so a callback that outlived an event-loop change raised "task_done() called too many times" or undercounted the new loop's queue. It now settles the queue the task came from. The rest are test isolation fixes for failures that only appeared when another file ran first on the same xdist worker: a replaced user_api_key_cache, breaker metrics unregistered by prometheus tests, semantic_router's health-check filter on uvicorn.access, logging tasks carried over from bedrock tests, a Router-written model_cost entry, and a stray post captured by the langflow test. The token counter check now asserts bounded chunking instead of wall-clock time. * test(e2e/ui): wait for the logout redirect before visiting a protected page Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED). * test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests * test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read * test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect * test(integration/mcp): answer the model-info probe another test's proxy sends to the model double |
||
|
devin-ai-integration[bot]
|
1f77fa65c8
|
fix(cost-map): registry audit 2026-09-26, MAI-Image-2.5-Flash price, Databricks Claude Opus 5.5, Azure Foundry retirement dates (#43254) | ||
|
devin-ai-integration[bot]
|
90873c46de
|
refactor(rust): expand logging and test coverage across gateway and Anthropic messages (#43295)
* refactor(rust): prepare inference and auth foundations * fix(rust): keep textract operations parsing from kebab-case model names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * done * refactor(types): derive Anthropic beta string conversions with Strum * fix(anthropic): report missing max_tokens as a missing field * refactor(rust): type Anthropic messages headers and auth after the Python layout Delete anthropic/messages/headers.rs. Its OAuth handling, credential ladder and beta merging move to anthropic/common_utils.rs where Python keeps them (optionally_handle_anthropic_oauth, get_auth_header, _merge_beta_headers), and the feature beta injection becomes update_headers_with_anthropic_beta on the messages config, as in Python. The BaseAnthropicMessagesConfig impl is unchanged apart from the bodies of validate_environment and request_headers Beta values are now the AnthropicBeta enum and BetaSet, which sort, dedupe and comma-join by construction. Request params gain typed speed, tools and context_management through Recognized, so the beta logic matches on enums instead of string-comparing JSON. OauthToken parses the sk-ant-oat token once and the chat config shares that detection instead of its own copy Case-insensitive header helpers move next to has_header in litellm-http. One deliberate divergence: a Bearer-prefixed OAuth key configured through api_key or ANTHROPIC_API_KEY is sent with a single Bearer scheme, where Python would emit "Bearer Bearer" Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * done * fix(rust): repair test compilation and clippy failures resolve auth before building the outbound request in prepare tests, give the host hook tests their own error type, and drop the disallowed reqwest client and err().expect() from core tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
99655b6f86
|
test: finish the non-proxy half of tests/test_litellm (#43281)
* test: move key-gated tests/test_litellm SDK tests into tests/llm_translation and drop empty folders * test: make token counter and health check unit tests run offline * ci: point unit shards, rust path filter, Makefile and docs at tests/unit * docs: fix stale test_litellm run paths in moved llm_translation tests * fix: correct databricks e2e sys.path depth and contributing example path --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d08746feb1
|
feat(proxy): email alerts at configured percentages of a team member budget (#42665)
* feat(proxy): email alerts at configured percentages of a team member budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): label team member budget crossings as team member budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover the team member alert dispatch from _check_team_member_budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(email): drop the emoji from the team member budget alert template
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): ignore team member alert thresholds outside 1 to 100 on both the backend and the dashboard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound team member alert threshold key length before int parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the legacy covers marker from the team member alert test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team): reject malformed team_member_max_budget_alert_emails on team writes
Thresholds outside 1-100, non-list recipients, and invalid emails now return 422 on
/team/new, /team/update and PATCH /team/{id} instead of being stored and silently
ignored. The value is stored canonically. Read-side LiteLLM_TeamTable is unchanged,
and the PATCH body stays a raw merge patch so a null threshold still deletes it.
* fix(auth): enforce and alert on team member budgets only in common_checks
The builder re-checked the team member budget inline before common_checks ran the same
check, so one request that crossed a team_member_max_budget_alert_emails threshold
dispatched two alerts. Drop the inline check; common_checks is the single authorization
point and already covers per-member rows, the team default member budget, zero-cost
skips and the cross-pod spend counter. Its 422 message now uses the TeamMember=user:team
form the builder and budget reservation already returned.
* Revert "fix(team): reject malformed team_member_max_budget_alert_emails on team writes"
This reverts commit
|
||
|
yuneng-jiang
|
2530255624
|
test: stop CI tests from downloading tokenizer files and images (#43257)
* test: load the embedding base image from a committed 100x100 PNG instead of downloading it * test: move the volcengine embedding test into tests/unit * test: check gpt2 and r50k_base tokenizer parity against committed tiktoken reference files * test: check hub tokenizer selection against an in-memory Hugging Face hub * test: serve image URLs from respx in the gemini tool-result and format-param tests * ci: drop the emptied legacy core-utils test path * test: cover the cohere and anthropic tokenizer paths in the hub tokenizer test * test: fetch every format-param image through respx and check its bytes reach the request * test: drop the gpt2 and r50k_base parity tests, which no litellm path uses * test: drop comments that restate assertions in the format-param test |
||
|
devin-ai-integration[bot]
|
4179860a17
|
fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) | ||
|
yuneng-jiang
|
7fc2206171
|
test: fix stale and state-leaking tests red on scheduled CircleCI (#43266)
test_update_config_success_callback_normalization replaced proxy_server.proxy_logging_obj with a MagicMock and never restored it. Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping, end-user and MCP tests on the same xdist worker awaited that mock and failed. The test now uses monkeypatch. test_prometheus_logging_callbacks set verbose_logger to DEBUG and litellm.set_verbose at import, so every worker in the unit job ran with DEBUG on. That broke caplog equality in the JEV classifier test, the vertex streaming memory ratio, and four event-loop lag checks. The module-level setup is removed; nothing in the file depended on it. #43081 removed the OCR harness modules but left them in the importability parametrize list. test_get_model_info_bedrock_region reassigned litellm.model_cost and set LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared the get_model_info caches, so it failed whenever an earlier test had looked up the regional model. It now uses monkeypatch and invalidates the caches; the local_testing isolation fixture also invalidates them after restoring model_cost. The Windows job hit CircleCI's 10 minute no-output limit while cargo compiles the Rust crates inside uv sync and uv build. Those two steps now allow 30 minutes of silence. |
||
|
devin-ai-integration[bot]
|
4adbc13d79
|
fix(router): hold Responses lifecycle events until output so a pre-output fallback announces one response (#43238)
* fix(router): hold Responses lifecycle events until output so a pre-output fallback announces one response * fix(router): narrow the responses wrapper close guards to Exception and test the hold helpers directly * test(router): type the responses fallback test helpers * fix(router): replay the held lifecycle events when the fallback stream fails before its first event --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e9491d31b5
|
refactor(rust): move credential inheritance and the SDK limits out of the legacy callback crate into a driver preflight (#43259)
Some checks are pending
Unit Tests: Documentation Validation / documentation (push) Waiting to run
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / mcp-integration (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
The legacy callback crate carried two rewrites that have nothing to do with the Logging contract: litellm_credential_name inheritance and the max_budget and num_retries_per_request checks. Any later callback host would need them unchanged, which is the smell the crate's AGENTS.md now names. They are now a Preflight the driver in litellm-host-python runs on the keyword view begin returned, before the host projects from it, supplied by python-bridge and passed through run_legacy_call. The call order is unchanged (setup, deployment hook, credentials, limits) and a rejection still fails the call as a host failure, so the failure callbacks run as before. The preflight rewrites the adapter's own copy in place, so no extra dict copy and no new lifecycle method Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
a11a93f44a
|
test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: move tests/test_litellm integrations and secret_managers into tests/unit Rename-only. Mirrors the old paths, including the directory conftests and the prompt and JSON fixtures. Follow-up commits prune and wire them. * test: prune and repoint the moved integrations tests Deletes the 7 audited tests a stronger test in the same tree already covers, imports the TLS sink helpers from their new conftest path, and restores os.environ after each integrations test. Some presets write OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the legacy tree's test ordering that header leaked into the AgentOps tests. * ci: run the moved integrations tests under their legacy flag The integrations GHA shard and a new CircleCI job run the integrations unit selection. secret_managers joins the misc selection. * docs: point integrations and secret_managers references at tests/unit * test: make the moved integrations directories packages * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit Rename-only. Mirrors the old paths, including fixtures, the stubtest config and the native-route wheel script. Two files that collide with existing unit files are merged in a follow-up commit. * test: merge, prune and repoint the moved core, routing, responses, caching and rust_bridge tests Merges the two files that collided with existing unit files, folding the legacy extra case into test_is_chat_completion_cached_dict, and deletes the 9 audited tests a stronger test in the same file already covers. Keeps what needs the network in tests/test_litellm: test_tokenizers pulls a tokenizer from the Hugging Face hub, and the gpt2 and r50k_base tokenizer cases download their BPE files. The unit core_utils conftest points TIKTOKEN_CACHE_DIR at litellm's bundled encodings so the rest never depend on import order to stay offline, and FakeSecretVault moves to a shared module so both trees can build it. * ci: run the moved core, routing, responses, caching and rust_bridge tests under their flags core_utils gets a core-utils flag and CircleCI job, and its GHA shard keeps the legacy path for the retained network tests. router_utils and router_strategy join enterprise-routing, responses joins responses-caching-types (minus responses/mcp, which mcp-integration owns), caching joins caching-local and rust_bridge joins misc. The redis-compat, test-rust, stubtest and merge-smoke paths follow the move. * docs: point the Rust crate references at tests/unit * test: make the moved core, routing and rust_bridge directories packages * test: keep the no-loop DualCache batch_get_cache regression test It runs the sync path outside any event loop, which the inside-loop test cannot, so a change that picks the Redis client by loop state would only show up there. * test: keep the job's UNIT_FLAG out of the shard-script tests * fix(url_utils): block 192.0.0.0/24 on every Python patch release * test: move the new budget limiter tests into tests/unit/router_strategy * test: move the new sentry scrubbing tests into tests/unit/litellm_core_utils * test: move the new zerobus tests into tests/unit/integrations * test: make tests/unit/integrations/zerobus a package * test: load litellm's own tiktoken cache setup once instead of resetting it per test --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e0fb89bc82
|
fix(proxy): keep the submitted body out of 422 validation errors (#43231)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
474ab91c09
|
test(zerobus): move tests into active CI selection (#43235)
* test(zerobus): move tests into the active CI selection * test(zerobus): add package marker for unit test discovery |
||
|
devin-ai-integration[bot]
|
d86c2e1f42
|
fix(logging): redact raw_request when turn_off_message_logging is set in the proxy config (#43219)
* fix(logging): redact raw_request when turn_off_message_logging is set in the proxy config The raw request branch bound turn_off_message_logging by name at import, before the proxy config set it, so loggers kept receiving the prompt in metadata.raw_request and raw_request_typed_dict. It now runs the same per-request redaction check messages use, and json_logs is read at call time for the same reason * fix(logging): keep raw_request_typed_dict for the explicit readers and tolerate missing headers in the json debug log * refactor(logging): drop the stale comment above the raw request typed dict --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Refael Iliaguyev
|
e065a2575b
|
fix(proxy): send a real error event when a /v1/messages stream fails (#41826)
* fix(proxy): send a real error event when a /v1/messages stream fails
When a stream failed halfway through, the proxy wrote the error as a plain
`data: {"error": ...}` line with no `event:` in front of it. Anthropic clients
pick stream events by that name, so they skip the line and the request looks
like it simply stopped with nothing in it.
Write the failure as an `event: error` frame with Anthropic's own payload, and
take the error type from the status code
* fix(proxy): use the shared Anthropic error mapping for the stream error frame
The first pass added a third copy of the status to error-type table, and it
disagreed with the documented one: 529 came out as `api_error` rather than
`overloaded_error`, and 413 as `invalid_request_error` rather than
`request_too_large`, which hides the two failures a client can actually act on.
Drop that copy and put the frame builder next to the table litellm already
keeps in anthropic_interface/exceptions. The bridged adapter path was building
the same frame inline, so it uses the shared one now too
* fix(proxy): seal a torn SSE frame before the /v1/messages error event
An upstream that drops mid-frame leaves the client inside an open event,
so the error frame that follows is glued onto the torn data line and the
Anthropic SDK raises a JSON decode error instead of an APIStatusError.
Close the open frame with a ping event the SDK skips before writing the
error event, and add the e2e stream-cut edge with Bedrock, Anthropic
boundary, and Anthropic mid-frame legs.
* fix(proxy): keep the SSE tail unchanged on a chunk that is not text
A serializer that hands a dict or model object through as-is has no bytes
the frame tail can learn from, so advance_sse_tail leaves it alone instead
of slicing it.
* fix(proxy): answer a /v1/messages stream that fails before its first byte as a JSON error carrying its status
* test: move the Anthropic error frame tests into tests/unit
* test(e2e): cut the upstream stream only after content has been relayed
* test(e2e): carry split SSE lines across chunks and always tear a data line mid-frame
* test(e2e): find the next data line across a chunk boundary before tearing it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
8ef85a45ce
|
feat(xai): add native xAI batches and files support (#42812)
* feat(xai): add native xAI batches and files support Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(xai): tighten batch handler typing and avoid Final redeclaration on star import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(xai): walk batch result pages iteratively Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(xai): stop paging on empty pagination token and honor litellm.xai_key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(xai): import NotRequired and TypedDict from typing_extensions for Python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(batches): accept image and video endpoints on batch create * test(xai): lock batch endpoint, auth, and result contracts The batches test package collided with litellm/batches under pytest prepend, so the All Other Providers shard could not collect the new tests. * fix(xai): price grok batch usage at xAI's 20 percent batch discount * fix(xai): map not-found file reads to 404, bill batch reasoning tokens, and add 200k batch tier rates * refactor(xai): drop routine prose and move tests under tests/unit * fix(health): hand the resolved provider to list_batches in batch-mode health checks * test(xai): make tests/unit/llms/xai/batches a package * fix(xai): walk every page of the files list by pagination_token --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mubashir1osmani
|
8b68c3cd09
|
fix(vertex_ai): keep legacy bucket_name in credential resolution and add GCS_BATCH_BUCKET_NAME env var (#42803)
* fix(vertex_ai): map legacy bucket_name to gcs_bucket_name and add GCS_BATCH_BUCKET_NAME env var * refactor(router): keep legacy bucket_name as a credential field instead of a validator Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): pass the RAG corpus bucket to the file upload instead of hopping through GCS_BUCKET_NAME * fix(vertex_ai): accept existing_file_id in the RAG Engine store step so ingest() runs end to end --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b6fcd03848
|
fix(bedrock): surface a converse-stream 200 that decodes to no events as a 502 instead of an empty turn (#43213)
* fix(bedrock): surface a converse-stream 200 that decodes to no events as a 502 instead of an empty turn * fix(bedrock): quote the body head only when a stream decoded no events The leftover-bytes error keeps the byte and event counts, the content type and the request id but no longer quotes the first bytes of a stream that already decoded events, since that head is the start of a healthy stream and can hold model output. The anthropic_messages empty-stream warning no longer prints the request's model name. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1fd04abb92
|
fix(responses): fall back on pre-output stream drops, fail truncated streams, honor request_timeout (#43133)
* fix(responses): fall back on pre-output stream drops, fail truncated streams, honor request_timeout A native /v1/responses stream that drops before any output item now raises the router's fallback-eligible MidStreamFallbackError, so configured fallbacks retry the original input. A stream that ends with a clean EOF or a [DONE] marker but no response.completed, response.incomplete or response.failed event now raises litellm.APIConnectionError instead of ending as if it had completed: fallback-eligible before any output, an explicit error after partial output. The sync iterator mirrors every branch. resolve_llm_passthrough_timeout now consults an explicitly set litellm_settings.request_timeout right after the router timeout and before general_settings.pass_through_request_timeout, so the router's native responses path honors it. * test(responses): give the normal-completion stream tests a terminal event --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
25de1ab2b0
|
fix(tests): drop the repeated UNIT_FLAG key in test_unit_shard_missing_paths (#43212)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
6b7688869e
|
feat(mcp): configure protocol versions and capability discovery (#43169)
* feat(mcp): configure protocol versions and capability discovery * test(mcp): return SDK initialization result in REST pagination fixture * test(mcp): arm cancellation deadlines after TCP calls start * fix(mcp): avoid serialized discovery and listing spend logs * test(mcp): isolate default protocol header policy * fix(mcp): honor preview protocol pins and refresh migrated tests * fix(mcp): preserve edited protocol pins in saved OAuth previews * fix(mcp): retain saved protocol pins when previews omit versions --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
cf491d1df9
|
test: move tests/test_litellm integrations and secret_managers into tests/unit (#43194)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: move tests/test_litellm integrations and secret_managers into tests/unit Rename-only. Mirrors the old paths, including the directory conftests and the prompt and JSON fixtures. Follow-up commits prune and wire them. * test: prune and repoint the moved integrations tests Deletes the 7 audited tests a stronger test in the same tree already covers, imports the TLS sink helpers from their new conftest path, and restores os.environ after each integrations test. Some presets write OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the legacy tree's test ordering that header leaked into the AgentOps tests. * ci: run the moved integrations tests under their legacy flag The integrations GHA shard and a new CircleCI job run the integrations unit selection. secret_managers joins the misc selection. * docs: point integrations and secret_managers references at tests/unit * test: make the moved integrations directories packages * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
5e6dc89ba1
|
test: move tests/test_litellm/llms into tests/unit/llms (#43191)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
f6882246d4
|
test: move tests/test_litellm root and small trees into tests/unit (#43186)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ccf866c801
|
fix(router): await budget redis pipeline before sync reads (internal copy of #32618) (#43125)
* fix(router): await budget redis pipeline before sync reads * refactor(router): remove superseded Redis flush helper * fix(router): preserve concurrent spend during Redis synchronization * fix(router): type per-key spend totals without loop Final bindings * fix(router): log Redis failures before cancellable cleanup * fix(router): finalize Redis batches before propagating cancellation * test(router): reproduce cancellation while Redis cleanup is blocked * test: align budget hotpath checks with awaited Redis flush * fix(budgets): finish Redis flush after cancellation while queued * test(budgets): consolidate Redis regressions in mapped tests * fix(budgets): coalesce failed Redis increments by key * test(budgets): assert spend behavior instead of batch state * perf(router): sum pending budget spend by key once per sync --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
020e5dee9b
|
fix(anthropic): keep the replayed prefix byte-stable for preserved thinking on chat completions (#42630)
* feat(anthropic): placement policy for mid-conversation system messages Pure functions over the OpenAI-format message list: split off the leading system run, keep later system messages as role=system at a placement Anthropic accepts on models flagged supports_mid_conversation_system (after a user turn, before an assistant turn or the end, never adjacent), and convert them to user turns in place elsewhere, keeping tool_result first in a merged user turn. * fix(anthropic): keep mid-conversation system out of the chat completions system prompt translate_system_message hoisted every role=system message, at any index, into the top-level system block. On a conversation carrying a mid-session reminder that rewrites the cached prefix, so the provider re-bills the whole history at cache-write pricing on every turn (#36559). #36968 fixed this on /v1/messages; the chat completions path, shared by first-party Anthropic, Vertex, Azure AI and Bedrock Invoke, still hoisted. Only the leading system run becomes the system prompt now. Later system messages go through the placement policy, and anthropic_messages_pt emits a system message instead of rejecting the role. The caller's message list is no longer mutated. Tests pin the two-turn prefix invariant across all four chat configs and both flag states. * refactor(anthropic): single-source the converted system note The /v1/messages pass-through and the chat completions path must prefix a converted system turn with the same operator note. * test(e2e): prove the prompt cache survives a mid-conversation system reminder on chat completions Same priming and assertions as the /v1/messages cases, through /v1/chat/completions with OpenAI-format messages, for first-party Anthropic and Bedrock Invoke on a flagged (Opus 4.8) and an unflagged (Haiku 4.5) model. The reminder sits between the assistant turn and the next user turn, the shape OpenAI-style agent frameworks send, which is the placement the chat path has to translate. * test(anthropic): cover the cache_control rebuild shapes and type the test helpers Codecov flagged the 5m ttl branch and the empty-system path of the wire builder; both now have a test. Greptile asked for full typing on the new test helpers. * refactor(anthropic): read the mid-conversation flag through a public supports_ helper supports_mid_conversation_system joins the other supports_* helpers in litellm.utils, so the chat transformation stops importing the private _supports_factory. * chore(typing): declare the mid-conversation type aliases with TypeAlias The Final sweep tightened LIT010, which exempts TypeAlias declarations but counts a bare alias assignment as an unannotated binding. * fix(anthropic): let add_code_execution_tool take the pass-through message union The translator now emits role=system inside messages for models that accept it, so anthropic_messages_pt returns the pass-through union. add_code_execution_tool still declared the narrower user/assistant union while only ever reading content, so upstream's strip_advisor_blocks_from_messages call in between made the mismatch visible to the type checker. * fix(bedrock): keep mid-conversation system messages in place on converse path * fix: ruff format + multi tool_result order + regression test * fix: satisfy type-discipline gate + update osv ignore for mlflow PYSEC-2026-3865 * fix(bedrock): restore role narrowing in hoisted system loop for basedpyright budget * test(bedrock): cover mid-conversation system conversion branches - non-dict guard in _opens_with_tool_result - in-place conversion without tool context - str/list cache_control preservation in mid-conversation path - drop unreachable non-system guard in hoisted loop * Place type-discipline suppressions on the lines the gate scans * Narrow hoisted loop to system role so basedpyright sees the right TypedDict * fix(anthropic): place mid-conversation system runs by their neighbours only A run after an assistant turn now slides behind the user turn that immediately follows it, and a run that ends the array or precedes an assistant turn becomes a user turn in place. No later message can move an earlier run, so a client that replays the conversation with more turns appended sends a byte-identical prefix and preserved thinking blocks keep their binding * refactor(bedrock): share the converted system note with the anthropic module Converse imports CONVERTED_SYSTEM_NOTE instead of carrying its own copy of the same text, and the reordering helpers lose their comments * test: pin the replayed request prefix across preserved-thinking turns One test per audited feature, through the real entrypoint: the chat transformations for anthropic, bedrock invoke, vertex and converse, the modify_params dummy tool result, dotprompt with unchanged variables, and Presidio masking against an in-process fake. Each serializes system, tools and the earlier messages of turn N and N+1 and asserts they match. The e2e mid-conversation system test imports its content blocks from models.py again and is marked provider_live * fix(anthropic): move mid-conversation system placement into prompt_templates The prompt factory imported the placement helper from the Anthropic provider package, whose common_utils reads a factory constant at import time, so loading the factory first raised ImportError. The module now sits next to anthropic_messages_pt and every consumer imports core utils A user turn with content [] or None puts no block on the wire, so a system run anchored to it landed first in messages or behind an assistant turn. Such a run now converts in place; empty strings and empty text blocks still anchor because the factory fills them with a placeholder * fix(anthropic): anchor system messages only on user turns that reach the wire * fix(bedrock): type the converse system-message helpers over the message TypedDicts * fix(anthropic): read replayed pydantic messages in the Converse helpers and convert a system run whose assistant follower sends nothing A history that replays the previous turn as the litellm.Message object was invisible to the Converse system-message helpers, so a mid-conversation system stayed between a tool call and its result or reached Converse as role: system. The helpers now read fields through the shared message_field and parts_of accessors and drop the local role predicate. Flagged placement anchored a system run on any assistant follower, but anthropic_messages_pt drops an assistant turn that puts no block on the wire (content None, an empty list, an unsigned thinking part), so the system landed directly before the next user turn, which Anthropic rejects. Such a run now converts in place. An empty or whitespace text turn still anchors, since the converter pads it with a placeholder. * fix(anthropic): treat bridged encrypted reasoning as a vanishing assistant turn for system placement An assistant turn whose only blocks carry Responses API encrypted reasoning is dropped by anthropic_messages_pt, so a mid-conversation system run anchored before it landed directly before the next user turn. The unsignable-thinking predicate now lives in common_utils and both the factory and the placement policy consult it. * fix(anthropic): let an inline thinking part hide separate thinking_blocks in system placement anthropic_messages_pt skips an assistant turn's separate thinking_blocks as soon as its content list carries an inline thinking or redacted_thinking part, so a turn whose inline part is unsigned puts nothing on the wire even when the separate block is signed. The placement policy now mirrors that rule. --------- Co-authored-by: Shifat Islam Santo <shifatislamsanto764@gmail.com> Co-authored-by: ege-arhan <egearhany@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f61b3c3f38
|
refactor(types): declare litellm-owned kwargs as typed objects and derive the lists from their fields (#42843)
* refactor(types): declare litellm-owned params in one registry * refactor(types): re-export registry constants without redundant aliases * refactor(types): satisfy type-discipline rules in registry projections and tests * refactor(types): classify every registry entry and check groups against typed config models * test(types): pin load-bearing names and exact projections in registry tests * style(types): keep agentic projection comment within ruff format * refactor(types): declare litellm-owned params as typed objects and derive the lists from their fields * refactor(types): fields of the typed objects become the registry; tests use a hand-written inventory * refactor(types): split traversal into wire_names and owned_wire_names, move rust to kwarg artifacts rust is a module-level switch (litellm.rust) that nothing reads from a call's kwargs, so it joins self, use_client and model_config as a registered artifact instead of a DispatchOptions field. The field constants now import from litellm.types.litellm_params directly instead of through a re-export in litellm.types.utils. metadata and litellm_metadata are MutableMapping because their readers mutate them in place, and client accepts raw httpx clients * refactor(types): own max_agentic_loops as an option and walk only nested leaves Move max_agentic_loops from AgenticLoopState to a new AgenticLoopOptions leaf under LiteLLMOptions, since the interception handlers read it as a deployment ceiling rather than stamping it. Drop the owned_wire_names fallback that treated an unresolved annotation as a direct field, which under postponed annotations silently shrank the registry. Re-export TRUSTED_CALLBACK_VARS_FIELD and ADDRESSED_RESPONSE_ID_FIELD from types.utils so that import path keeps working. Tests use hand-written inventories for the callback and pricing names * refactor(types): move data_residency to call state and drop aliased re-exports data_residency is stamped by get_litellm_params and responses.main during the call, so it lives on CallState, not CostOptions. mock_response also accepts a float sequence, which main.py reads for mock embeddings. The types/utils.py re-exports become one plain import with an exact F401 suppression instead of two X as X aliases that pushed PLC0414 over its strict-gate ceiling. Redundant leaf docstrings and the structural artifact test are gone; the re-exported FIELD constants are checked by identity instead * refactor(types): project owned kwarg names once and keep pass-through extraction in request order * refactor(types): type caching_groups from its cache reader and hoist the pass-through ownership set caching_groups is a sequence of flat model-group sequences, which is what Cache._get_caching_group iterates. A regression test drives the public cache key path so two groups in one caching group share a key and a third does not. The pass-through endpoint builds its frozenset of owned names once at import instead of per request, reads the two metadata carriers from the extracted mapping instead of popping them, and its extraction mappings are read-only. Concatenation tests assert the whole derived list and tuple, docstrings drop reader claims that nothing in the module backs * refactor(types): read owned names live in pass-through and pin tests to literal inventories The pass-through endpoint checks body keys against the public all_litellm_params list at request time again, as the base does, instead of a frozenset taken at import, so a name registered after import is still extracted. A test drives that path with a name added after import, and another sends both metadata carriers interleaved with provider keys and asserts the whole merged result. retry_policy accepts the mapping form its router reader builds a RetryPolicy from. The pricing inventory in the typed tests is a literal tuple checked against the model's fields, the agentic compatibility test asserts type, length and set instead of declaration order, and the typed-model overlap tests assert the exact intersection. * refactor(types): move model_alias_map to CallState and read the owned registry in registry order in pass-through * refactor(types): drop restating docstrings, keep FIELD importers on types.utils, pin pass-through registry order * fix(types): satisfy strict lint for public FIELD re-exports * fix(types): restore clean parameter re-exports * fix(tests): compare pass-through extraction order to registry body keys * refactor(types): type owned request parameter leaves * refactor(types): share routing strategy literal and tighten leaf tests * fix(proxy): drop client-supplied proxy-stamped names from pass-through litellm_params * refactor(proxy): name pass-through litellm key split for what it holds * refactor(types): drop TODO markers on the kept readerless fields * fix(types): keep deployment tag_regex and max_file_size_mb out of provider requests * fix(types): include every routing strategy the router accepts --------- Co-authored-by: shrey kharbanda <shreshth@berri.ai> |
||
|
devin-ai-integration[bot]
|
d0d3b6a67e
|
fix(vertex_ai): stop advertising OpenAI platform-only params on Gemma and Llama routes (#43079)
* fix(vertex_ai): stop advertising OpenAI platform-only params on Gemma and Llama routes The Anthropic /v1/messages bridge derives prompt_cache_key from Claude Code's session id whenever the provider config advertises it, and every Vertex OpenAI-compatible route (gemma/, openai/<endpoint>, meta/) inherited the full OpenAI list, so the Model Garden vLLM container rejected each turn with a pydantic extra_forbidden 400. Vertex's Llama and Gemma configs now filter one shared list of platform-only params (prompt_cache_key, prompt_cache_retention, safety_identifier, service_tier, store, web_search_options, modalities, prediction, audio, max_retries) out of their supported params, so the bridge no longer derives the key and drop_params drops an explicit one. * fix(vertex_ai): scope the platform-param filter to self-deployed Model Garden endpoints --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5a75f09d6d
|
fix(vertex_ai): surface the Gemma container's own error inside a 200 :predict response (#43075)
* fix(vertex_ai): surface the Gemma container's own error inside a 200 :predict response * refactor(vertex_ai): move the gemma container error parser next to its adapter --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
25fb7810c2
|
fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token (#40541)
* fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token A CLI session token is a fresh random secret on every login, so since v1.99 each login's spend rows carried a different sha256 hash as api_key and the usage APIs could resolve neither key_alias nor user_email for them. Spend rows and logging callbacks now attribute a session request to its stable alias, cli-session-<user_id>, and the usage endpoints derive that alias and owner from the key itself instead of scanning for a matching digest * fix(spend): resolve the CLI session team from the user's first team in usage metadata A cli-session key carries no team of its own in the DB, so the usage breakdown showed team_id None for it and the export grouped it as Unassigned. The login attaches the user's first team to the session, so the recovery mirrors that rule for cli-session keys only. * fix(spend): claim the session team only for a single-team user The CLI login attaches a team on its own only when the user has exactly one; a user in several teams picks one per login, so usage metadata for the alias would otherwise name a team the login may not have used. * test(pass_through): mark the mocked auth object as a plain key The logged key follows the alias only for a session token; a bare MagicMock reads as one, so the test names the field it relies on. * fix(spend): attribute CLI session pass-through, queue, and managed batch spend to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): only treat the exact cli-session-<created_by> value as a batch key alias A managed object row written by an older build can still carry the raw per-login session token, which shares the cli-session- prefix. Matching on the prefix alone would have surfaced that token as a trusted alias and persisted it verbatim in the batch cost spend log, so the alias check now requires the exact per-user value and every other prefixed value keeps going through redaction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): log proxy executed batch rows under the cli-session alias instead of the session token _row_metadata set user_api_key from the raw bearer token while user_api_key_hash carried the alias, so the spend log redaction rejected the alias as untrusted and hashed the random session token instead Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): attribute semantic search embedding spend to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): scope /key/spend/report for a CLI session to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): use the cli-session alias for websearch spend, prometheus failure labels and the parallel limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(spend): drop explanatory docstrings on get_logged_api_key and attach_user_details Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): only recover cli-session usage keys whose suffix is a known user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e2302be068
|
refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081)
* refactor(ocr): remove the Python OCR execution path and require the Rust route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fmt * refactor(ocr): tidy the native OCR passthrough binding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ocr): ruff format the azure passthrough transformation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ocr): resolve passthrough OCR costing in one Rust call Replace passthrough_url/passthrough_transform with passthrough_response, which matches the relayed endpoint against each Azure config's path segments instead of building a fake request to call get_complete_url. The binding drops the unused headers, status and api_base arguments. Catch the ValueError/RuntimeError the binding raises so a relayed body that is not OCR-shaped falls back to the passthrough object instead of failing logging, and cover the relay against the real binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ocr): drop the unused LlmProviders import from health check helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: drop the ocr_testing job now that tests/ocr_tests is gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ocr): restore the live OCR matrix and the ocr_testing job The public litellm.ocr / aocr / Router interface is unchanged by the Rust migration, so the live provider matrix still applies. Drops the stale VCR skip list for the deleted test_rust_bridge.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ocr): import Final in the health check helper tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
a76f23ac4f
|
fix(bedrock): map Anthropic batch row params the way real time does (#43087)
* fix(bedrock): map Anthropic batch row params the way real time does * fix(bedrock): let a batch row's allowed_openai_params reach the mapper * test(bedrock): assert the batch thinking value matches the real-time mapping * fix(bedrock): keep json_mode out of Anthropic batch rows and pin route-prefixed deployments --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e450f2d54c
|
fix(router): serve Responses turns from a sibling when the encrypted content origin has no boundary peer (#43015)
* test(integration): reproduce encrypted_content_affinity 503 when origin has no boundary peer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep encrypted content affinity turn one out of the response cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): degrade encrypted_content_affinity when the origin has no encryption-boundary peer A routed-group candidate that is currently unavailable and shares its (api_base, api_key) with no healthy deployment used to raise a proxy-level 503/429 from _unavailable_origin_error, even though healthy siblings in the same model group could still serve the turn. Strip the encrypted reasoning and dispatch to the healthy pool instead, matching the existing cross-group behavior, and log a warning naming the origin model_id and routed group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): bound the degraded-affinity log marker and assert the strip on the wire Address review findings: restore num_retries alongside optional_pre_call_checks in the integration test teardown, record scenario request bodies on the scripted upstream so the tests can assert no encrypted reasoning reaches the sibling, and truncate the client-supplied model_id in the degraded-dispatch warning Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): only record JSON bodies on the scripted upstream Multipart uploads to scripted POST routes have no JSON body, so gate the observation recording on the request content-type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): make encrypted_content_affinity runtime-toggleable so /config/update can turn it off --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3fa688223d
|
fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge (#43042)
* fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge Vertex batch uploads treated every non-embeddings JSONL row as a chat completions body, so a /v1/responses row lost its input and reached GCS as a blank text part. Route detection now recognizes /v1/responses rows and bridges them to chat through the same Responses-to-Chat bridge the real-time path uses. That bridge call moves out of the Bedrock files transformation into a shared helper both providers call, forwarding the record's fields as sent, like real time, instead of validating them against the SDK TypedDicts whose required keys clients omit. * chore(batches): type the Vertex responses test helper and drop the quoted input cast * fix(batches): translate developer messages to system on Vertex and Bedrock batch rows like real time --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ba77646991
|
ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests (#42904)
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests * ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * build: point the local proxy unit targets at the nested tests/unit/proxy tree * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai> |