mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tessellark
|
b5a72c2042
|
fix(router): pin /v1/responses requests for prompt caching affinity (#50321)
* fix(router): pin /v1/responses requests for prompt caching affinity * fix(router): pin previous_response_id turns and keep native tools in the Responses pin key A Responses turn that continues a stored response now goes to the deployment that answered it, unless that deployment already failed this request. Native Responses tools are hashed into the pin key as sent instead of converted, so two requests with different native tools no longer share a pin and a native request no longer logs a false dropped-tool warning. The pin keeps its cacheable deployment ids in a sorted tuple so loggers can serialize request metadata. * fix(router): leave a Responses request with a malformed tool to the provider * fix(router): keep the Responses prompt cache pin out of logged metadata The pre-call check now keeps the pin in a router-owned in-flight store and puts only a random token in litellm_metadata, so loggers that ship metadata never see the routing key or deployment ids. The store is shared across check instances, so turning prompt_caching off and on at runtime keeps pinning, and a retry drops its previous attempt's token. The skipped-input debug log now names only the error type. * fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries The prompt caching check no longer narrows the candidates inside the callback filters. It names the deployment it prefers, and the router core applies that preference at the pinned deployment step, after tag, region, pre-call and plugin filters, only when the deployment is still a candidate. A pin can no longer send a request to a deployment its tags do not allow. A retry skips the pin when the pinned deployment was already tried in this request, read from the retry breadcrumbs in both metadata buckets and the excluded and retry-skipped ids. When the retry count outruns the breadcrumbs the check stands down, so a pinned deployment that keeps failing is not retried on every attempt. * test(router): list the prompt caching check as an in-memory cache caller The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers * fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records * test(router): lift test cooldowns through the cooldown store's evict_local * fix(router): key Responses tools like their chat twin and drop the pin token after success A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict. * test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
067f93340f
|
refactor(router): delete the python router core, the rust core is the only core (#50178) | ||
|
Nate Armstrong
|
9752869d84
|
feat(router): add the Rust router core with the host as its only boundary (#49513)
Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/), served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in the host-side litellm/router_rust/ package (steps/, invoke/, surface/) driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state in litellm/router_rust/surface/ (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. The crate records each attempt's outcome itself rather than through litellm callbacks. A streamed attempt is deferred: it counts its request when the stream opens, and the host's stream wrappers report the success with the final usage, or a failure after the first chunk, when the stream ends, as the Python core's callbacks do. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores, with nothing pending; only the names of the Python core's own outcome callbacks are asserted on Python alone. litellm.Router is now final: subclassing it raises TypeError at the class statement, and the machinery that kept subclass overrides working is gone. The Rust core never honoured subclass overrides, so they would have silently stopped running there. Custom routing goes through register_plugin(), settings through apply() and outcomes through observe(). This is a breaking change for anyone subclassing Router. Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries, max_fallbacks, allowed_fails, retry_after, timeout, the tag settings, routing_strategy_args and the fallback lists as given (the crate reads them as numbers, update_settings casts as Python does); a deprecated `model_list = [...]` write reaches the Rust core; a failed set_model_list is all or nothing on both cores (the deployments served before the call are put back and the error raised); config_deployments() is a generator and `del` of a plain attribute works on the Rust core. Least-busy pick reservations from main now work on the Rust core: its selectors share one reservation store, routed calls reserve and free their unstarted picks, and select() and batch cancel do not reserve. On the Python core a dropped upsert takes its name back off deployment_names, and get_deployment_order reads every numeric spelling as one int so mixed orders no longer raise TypeError. Rulings on the remaining differences, the Python core changed where it was the odd one out: sync picks and picks by deployment id apply the team filter as async picks do, so another team's deployment is refused; the fallback lists and default_litellm_params in RouterSettings are copies, so an in-place edit is ignored and settings change only through apply(); one health record with an unreadable timestamp drops out alone; an empty fallback entry is skipped; a bool cooldown_time is refused at construction, a bool or string cooldown_time on a deployment or a call falls back to the router default, and a failure status that no HTTP status can be is recorded as 500; an infinite or NaN cost and a whole number wider than 64 bits (ptu_shares) are refused on both cores with the same error (litellm/router_utils/deployment_numbers.py); and a /v1/messages stream whose provider iterator raises cools the deployment and retries like a non-stream call, with the Rust core waiting for that cooldown before its retry picks. Differences the proxy parity fuzz and its regression tests found on the Rust core: a file upload or a batch on an uploaded file falls back only within its own model group, as on the Python core, since the file lives with one provider (the host sends same_group_fallbacks_only and the walk skips targets in other groups); Decisions calls drop deployments whose provider has no Decisions config, and deployments that refuse the audio or videos a call carries, unless that would leave none, as the Python core does, while the retry count still reads the whole group; a file upload that fails on one order level is created on the next level's deployments rather than the first level again; and a response that cannot be converted (a text_completion body without a message) reports only its own traceback, so a rate limit handled by the attempt before it no longer turns the error into a 429, and both cores return 500. |