mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
108 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yujonglee
|
2be96aed13
|
refactor(redis): delete the Python RedisBatch now that every request op is native (#51383)
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left |
||
|
yujonglee
|
9a1acee4a7
|
feat(redis): serve the router's session pins from a native store (#50870)
Deployment affinity and the complexity router's session and tier pins move off the router's DualCache onto RouterSessionPinStore, backed by the native redis-store. The read and claim Lua scripts run byte-identical to main, the pin read rides the request's pre-call batch, and keys and values keep main's format. |
||
|
yujonglee
|
b88e4bff6b
|
refactor(router): count the whole router_usage family on the native store (#50862)
* refactor(router): count the whole router_usage family on the native store usage-based-routing-v2, the model rate limit check (tpm, rpm, tpd, rpd) and the io-token check (itpm, otpm) count through the native router_usage store, next to the router core's global_router tpm and rpm. The pick's usage read rides the native pre-call batch and settles on the batch reply, keeping DualCache's per-key read throttle, and the success tpm increment rides the native post-call batch after counting in memory. io-token reservations stash their counter's fields instead of a Redis key. Redis keys and values are unchanged. A Redis service event is reported in a context named for whose it is, either the calling store op's, the request whose post-call batch its deadline sends, or nobody's for breaker transitions and unscoped calls. The deadline's tag is built when a post-call declaration arms it, so that flush parents under the request's span even though auth sets the span after the batch scope opens. An SDK call anchors the span its caller passed, so its usage reads and io-token refunds parent under it without a span argument threaded through the family. * fix(redis): keep an int counter past i64 above the limit instead of wrapping Number::add summed two ints with plain i64 addition. With Redis off and the OTPM counter at 1, a request with max_tokens=2**63-1 passed the MAX_INCREMENT check, and the memory count wrapped negative in a release build, so it passed admission and let other requests past the cap. Python's int never overflows, so main sees a count above the limit, rolls it back and answers 429 An int sum past i64 now becomes the nearest float, which compares above any limit the way Python's int does. The rollback after such a refusal can leave the count one token low, since the float cannot hold 2**63 exactly * fix(logging): run every callback under the span an SDK caller passed Native Redis service spans take their parent from the request span contextvar. The proxy sets it at auth and the SDK router scopes it around the routed call, but a streamed response is iterated after that scope closes, so its callbacks saw no span. Three callbacks reopened the scope from kwargs by hand, and the sync stream path had no wrap at all because its success handler runs on an executor thread The four Logging handlers now anchor the span from model_call_details around their dispatch, so every CustomLogger callback sees it, streamed or not, sync or async, with or without a Router. The per-callback wraps in lowest_tpm_rpm_v2 and model_rate_limit_check are gone. A span already anchored, as on the proxy, is kept * test(logging): cover a mid-stream failure, and drop the span wrap in Router._release The callback span test now also breaks a stream after its first chunk, sync and async, through a custom provider, so the failure handlers the stream wrapper starts on a bare thread are covered too A probe through the Router showed Router._release only runs inside the routed attempt, where the router has already anchored the span, and never after a stream returns. Its own wrap was a no-op, so it is gone * fix(router): refuse an io-token reservation past its limit before counting it The overflow fix let a reservation near i64::MAX become a float, so the count still read above the limit and the request got its 429. But the rollback subtracted the same amount from that float, and with OTPM usage at 40 the count came back as 0.0, so later requests reused capacity already spent A reservation above its limit now gets the 429 before the store is touched, in the async and sync paths. On main the count went up and straight back down for such a reservation, so the answer and the count after it are the same as main's. Only the usage named in the error message differs, which now shows the reservation instead of the count plus the reservation * test(redis): send the failed SDK call's refund through a real completion The refund span test called ModelRateLimitingCheck.async_log_failure_event directly, so it relied on the span wrap that callback used to carry. The Logging failure handlers now anchor the span for every callback, so the test sends a failing litellm.acompletion with the reservation in its metadata and checks the refund's Redis span still lands under the caller's span |
||
|
yujonglee
|
e20f07d178
|
feat(redis): serve the health checks' deployment states from a native store (#50357)
* feat(redis): serve the health checks' deployment states from a native store DeploymentHealthCache reads and writes its one Redis entry through a new deployment_health store family instead of the router's DualCache, like the other router families. * refactor(redis): type the deployment health states in the store The health store kept a raw Value from Redis to Python and back, so Python re-checked the shape by hand and router-core only got a typed record after a Python pass. The family now has a types.rs with HealthState and HealthStates, the bridge converts through PyData, and the Python store returns DeploymentHealthStateValue mappings Reads keep main's leniency for records another writer left: extra fields are ignored, a falsy flag is unhealthy and a missing one healthy, a timestamp that is no number reads as 0, and a record that is no dict is dropped alone. A rewrite writes only is_healthy, timestamp and reason |
||
|
Nate Armstrong
|
59538ff1d9
|
feat(messages): serve anthropic messages on rust only (#51119)
* feat(messages): apply modify_params rewrites on rust * feat(messages): mint anthropic workload identity tokens on rust * feat(messages): apply prompt caching injection on rust * feat(messages): serve the easy python-only detectors on rust * feat(messages): retry rejected thinking signatures on rust * feat(messages): serve response cache reads on rust * feat(messages): retry rejected strict tools on rust * refactor(messages): delete the python path for anthropic messages Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a detector that raises) gets UnsupportedParamsError naming the feature, raised from one place in litellm/messages/dispatch.py. Other providers keep the Python fallback. A call whose arguments the handler signature rejects still gets Python's TypeError. Deleted Python code that only served anthropic Messages: - workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses were never eligible; the Rust host mints the token itself) - the anthropic Foundry beta-row and affinity-base branch in async_anthropic_messages_handler, and the "forward" mode of the beta filter that only that branch used The proxy MCP tools detector now uses the light tool precheck so a function tool never imports the gateway handler, since a raising detector now refuses. * feat(messages): forward message threads on rust The Rust path for anthropic /v1/messages dropped the thread body field, so a create or continue turn reached Anthropic without its thread or the message-threads-2026-08-12 beta. It now matches the Python handler from #50976: - the host forwards thread with the other body params - a non-null thread adds the message-threads beta - a well-formed continue turn keeps advisor blocks in history and derives no compaction, per-turn-control or tool-change beta from its messages - the recorded httpx_response carries the request that was sent, for JSON and streamed calls, so the proxy's thread ownership hook resends the create turn's betas on a continue as it does for Python Upstream error.details already reached the exception message unchanged, so thread_not_found relays and keeps the deployment out of cooldown on both paths. Parity cases cover create, continue, streaming, sync, retries, thread_not_found and the ownership hook round trip. * feat(messages): serve mock responses on rust * feat(messages): send caller clients and unsupported tls through python on rust * feat(messages): run pre-request hooks on rust * feat(messages): run agentic loop hooks on rust * feat(messages): run the safeguards classifier on rust * feat(messages): run registered interceptors on rust * fix(messages): match python stream cache gate, logging and interceptor order on rust Store a streamed Messages response in the cache only when Python would: a message_stop event line, no provider error frame and at least one frame with output. Leave dict results untouched by mark_rust_response so hidden params never reach result dicts, logging objects or hook digests. Ask interceptors whether they handle the call after the cache lookup, not at decode time, so a cache hit never calls can_handle. Emit the track_llm_api_timing service event for the provider exchange. Bill and log a stream superseded by a router fallback as the failure of its error frame, with the partial usage warning. Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the autorouter baseline cache sees retried requests. * fix(messages): forward python-only json values to anthropic as python does Accept any max_tokens, sampling scalar and messages collection that Python main accepts, and carry values serde_json cannot hold (ints beyond 64 bits, NaN and infinities, strings with lone surrogates) as validated raw JSON leaves written back verbatim on the wire, so the upstream body matches Python's json.dumps * fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust * fix(messages): log python's plain provider header names on rust streams * fix(messages): match python's request params, logged params and validation on rust Read a non-bool stream by Python truthiness and send it as given, log the caller's stream flag as Python's handler does, raise the raw Python errors main raises for malformed messages, tools, model_info and hook-set litellm_params or providers, read the tool custom-field drop from drops_unsupported_params, log the caller's nested fields that additional_drop_params strips and the Python values of raw JSON leaves, keep a missing stop_reason absent, and map provider-config rejections (missing max_tokens, unknown reasoning effort) as Python's HTTP handler maps them, with Python's text, effort order, model and provider * fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust * fix(messages): match python on hooked streams, empty cache writes and mcp cache hits * fix(messages): join already buffered stream reads and trim the env api key as python does on rust * fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust A stream cut by a transport error now hands the caller the bytes already buffered before raising, as aiohttp does for a caller that reads them before the event loop sees the cut, so the frames and their usage reach the caller and the failure log. Provider replies keep integers wider than 64 bits exactly and read numbers that overflow a float as infinity, through the response, the response cache and the logged result. The provider client follows up to 20 redirects as httpx does, and running out raises Python's text. A provider_specific_header value the ascii codec cannot encode fails before the send like a caller extra header. The deployment success hook gets the stream as AnthropicMessagesStreamingResponse, as Python's handler hands it. * feat(messages): serve proxy mcp tools on rust * test(messages): cover hook, agentic and interceptor edge cases in the parity suite * fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust * fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust A header value Python's client cannot encode as ASCII now fails before the send whichever header carries it, so a caller api_key with a non-ASCII character raises Python's InternalServerError instead of a raw UnicodeEncodeError after the reply. A caller api_base without an http(s) scheme, or with whitespace, control or non-ASCII characters, is sent through Python's client at the URL Python builds, so its relative-URL and invalid-URL errors match, and a base urlsplit cannot split raises Python's bare ValueError before the send. A provider_specific_header of the wrong shape is scoped through Python's ProviderSpecificHeaderUtils first, so a plain string raises Python's AttributeError. * fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust temperature and top_p now deserialize through recognized::exact_float, which keeps an integer where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent upstream and logged as the integer python forwards, and comparisons still go through scalar.rs cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python TypeError marked as a late rejection, and map_failure hands it to the caller unmapped * test(messages): format the non-string tool_result parity helper * fix(messages): flag and log the call before python's pre-send raises on rust The bridge now hands settings() and map_failure() whether the call is async. A provider_specific_header of the wrong shape sets the aanthropic_messages flag before re-raising, as Python's handler has already flagged the call. A base urlsplit cannot split also logs the resolved model, provider and optional params first, as Python raises it only after its logger is updated. The TypeError raised while capping the thinking budget for a string max_tokens sets the flag too * feat(messages): serve the web search short-circuit override on rust through the intercept host call * fix(messages): give the web search entry kwargs its mutable-ok reason * test(messages): drop the body hook gap the rust short-circuit closes * fix(messages): name the partially unknown handler types on the short-circuit host call |
||
|
joshua-berri
|
82471d4603
|
fix(responses): keep reasoning item IDs consistent across streaming events (#51243)
* fix(responses): keep encrypted reasoning IDs consistent across stream events * fix(responses): preserve immutable stream events during affinity encoding * fix(responses): type affinity encoding input boundaries * fix(responses): use immutable affinity item updates --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
zoveki
|
a1f3dcd01b
|
fix(completion)!: answer 400 for a chat message without a role (#51124)
* fix(completion): answer 400 for a chat message without a role A message with no role was filled with role assistant and sent upstream, so the caller's text reached the model as its own earlier reply and the request returned 200. Now a missing, null, or empty role answers 400 with OpenAI's wording, naming the message index. A role-less turn that carries tool_calls is still sent as assistant, since only an assistant writes tool calls. * fix(completion): treat a role-less message as the caller's error The 400 now raises MalformedRequestFieldError with OpenAI's param and code and the provider name, so it no longer counts toward the deployment's failure rate, cools it down, or fires an llm_exceptions alert * fix(completion): keep the assistant fill for a role-less function_call turn * refactor(completion): type the role-less call turn check and the request kwargs Read role, tool_calls and function_call off the cleaned message copy, and validate completion's kwargs into a typed mapping before naming the provider on the 400 * fix(completion): keep the alert and failure-count exclusions to the role error Raise MissingMessageRoleError, a MalformedRequestFieldError subclass, so a malformed tools 400 alerts and counts as on main, and fill assistant whenever a role-less turn has a tool_calls or function_call key * fix(completion): keep assistant-only role-less turns and skip SDK and PagerDuty alerts A role-less turn carrying any assistant-only key, null values included, is still filled as assistant, so a dumped assistant message or a reasoning turn keeps working. The Router's alerting_config alert and the PagerDuty failure count skip the role error, like the proxy alert already does. * fix(utils): drop the string code from the missing-role 400 so the Errors tab counts it as 400 The spend log stored code missing_required_parameter as the error code, which the Usage page Errors tab cannot parse as a status, so every role-less request landed under no status. The malformed tools 400 leaves code out the same way. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
9ebbcc7710
|
feat(anthropic): forward message threads on native /v1/messages (#50976)
* feat(anthropic): forward message threads on native /v1/messages Keep the thread body field on native Anthropic legs, add the message-threads-2026-08-12 beta for it, reject it with a 400 on legs that cannot continue a stored thread, and relay error.details of an upstream Anthropic error so thread_not_found reaches SDK thread helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(anthropic): keep a stale thread 404 from cooling the deployment and type the thread beta ignores Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: sort the cooldown handler imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): dedupe the message-threads beta map rows and annotate test finals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): bind message threads to the key that created them and answer unsupported continues with thread_not_found Every virtual key on a proxy shares one Anthropic credential, so forwarding thread alone let any key continue another key's thread by its message id. A new default proxy hook binds each thread reply id to its virtual key, or to the signed-in identity when there is no key, and answers a continue naming an id it did not bind with the 404 thread_not_found the SDK replays on. Legs without message threads now serve a create without the thread and answer a continue with that 404 before any upstream call, instead of answering it with no history. * fix(proxy): keep a cached message thread reply with the key that got it first A response cache hit replays the original reply id, and binding it again handed the thread to the second key, so the creator's continue got thread_not_found. * fix(anthropic): keep message threads bound on every route and stable across turns Check the thread binding on every route that carries thread except pass-through, keep bindings in their own cache so they never evict rate-limit counters, add no tool or message derived betas on thread turns, answer a create on a leg without threads with a 400 thread_unsupported_request unless drop_params is set, and forward threads on Claude Platform on AWS * fix(anthropic): reject message threads on /v1/messages passthrough deployments A deployment that forwards /v1/messages untranslated to an OpenAI-compatible server inherited the native config's thread support, so `thread` reached a server that keeps no thread state. It now answers the same 400 thread_unsupported_request on create and 404 thread_not_found on continue as a translated leg * fix(anthropic): resend a thread's create betas on continues and check bound ids on pass-through A thread create now gets the betas LiteLLM derives from its tools and messages again, so an advisor tool create that answered 400 on the previous commit answers 200. Anthropic fingerprints the beta set, so the proxy keeps the anthropic-beta header the create sent upstream in the thread binding and adds it to every continue on /v1/messages. It reads that header from the upstream request, since success logging overwrites the logged request details before the success hook runs A continue on the /anthropic pass-through route of a thread the proxy bound to another key now gets the 404 thread_not_found. Ids the proxy never bound still pass through, as before * fix(anthropic): continue unbound threads off /v1/messages and keep every anthropic-beta on chat and Responses * fix(anthropic): skip non-string anthropic-beta list entries instead of raising * test(anthropic): keep the header merge regression test clear of main's appended tests * fix(anthropic): add a thread's create betas to the client's own on continues and keep base header merging on chat and Responses * fix(anthropic): answer an unbound thread id on chat and Responses with the 404 and drop the stale beta-join tests * fix(proxy): check a thread continue sent inside extra_body against its owner Azure AI promotes extra_body into the request body, so another key could continue a thread by putting thread there, past the ownership hook that read only the top-level field * fix(proxy): write a thread's first owner atomically and bind creates sent inside extra_body Two workers that both miss a reply's binding now agree on the first key stored in Redis, since the claim uses SET NX and each worker caches the stored winner. A create sent inside extra_body now binds its reply like a top-level create. The redundant Claude Platform supports_message_threads override is removed, since the provider already resolves to the native Anthropic beta list * fix(proxy): include agent_id in the key-less message thread owner * fix(proxy): tag message thread ownership Redis calls with their key family --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
8092edc7a1
|
fix(proxy): judge the zero-cost budget waiver by the model a key or team alias serves (#51058)
* fix(proxy): judge the zero-cost budget waiver by the model a key or team alias serves The zero-cost budget waiver judged the name the client sent, but key `aliases` and team `model_aliases` are applied later during request preparation. An over-budget key got a 422 for an alias of a $0 model, and an alias named like a $0 group that points at a paid model was waived and served past the budget. For a string body `model` on chat, completions, responses and messages, the key-path, centralized and custom-auth waivers now judge the model request preparation serves. That model comes from the same team, key, global and router alias functions that request preparation uses, so the two cannot drift. Other routes and the JWT path keep judging the requested model. common_checks no longer overrides a caller's explicit decision not to skip budgets, and proxy-made sub-calls are judged by their own model. Resolves LIT-10283 * fix(proxy): waive only once auth has applied the router aliases, and keep empty team alias targets The served-model waiver now applies only after auth's router-alias rewrite has run on the request body. Custom auth without common checks runs its waiver before that rewrite, so it keeps judging the requested model instead of a key or team alias of a name the router rewrites elsewhere. The waiver also keeps an empty-string team alias target, matching request preparation. Resolves LIT-10283 * fix(proxy): waive by the served model only for a client's own body-model request The waiver resolved aliases for any request on a chat, completions, responses or messages path, including the HTTP request the Responses WebSocket builds for first-frame admission. That session dispatches the model as an endpoint model, which skips the team alias, so a team alias of a paid model to a free one waived the paid model's budget. The waiver now resolves aliases only when FastAPI matched the request to one of the body model routes, which requests built inside the proxy and path model routes never are. The waiver tests are folded into one table, share a priced router fixture, and cover the stale team alias flag cache. Resolves LIT-10283 --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
e2625d0509
|
fix(router): re-check and reserve per-model rate limits on fallback targets (#50994)
* fix(router): re-check and reserve per-model rate limits on fallback targets A provider error fallback handed the caller the fallback model group without its per-model TPM/RPM limits ever being checked. The router gets a fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires it to the v3 limiter, which reserves the target model's per-model descriptors before each fallback hop and skips targets that are over their limit. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): dedupe a repeated fallback target before the reserving rate limit check Both walks now consult the attempted set before the rate limit check, so a chain like ["B", "B"] reserves B once. The deprecated settings parity test also learns the new setting. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): a bidirectional fallback chain never re-reserves the requested model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): match the surfaced failure in the fallback reservation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the admission stash untouched while building fallback descriptors Building the fallback target's descriptors could rewrite the request stash's PTU ceiling, which post-call settlement reads for the model the caller named. The fallback check now hides the stash while it builds, like the frame check. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
f0d234c4c6
|
fix(router): let an explicit routing tag outrank the encrypted content pin (#50975)
* fix(router): let an explicit routing tag outrank the encrypted content pin A Responses follow-up that replays encrypted reasoning minted on one deployment, but asks for another with a routing tag, failed with "Not allowed to access model due to tags configuration". The affinity check now runs the router's tag filter on the deployment it is about to pin, and on encryption-boundary peers. When the filter would reject them, it releases the pin and strips the reasoning the tagged deployment cannot decrypt, so the request routes there. Requests the tag filter accepts behave as before. Fixes #50926 * fix(router): strip reasoning against only the boundary peers the tags keep When tags rejected some encryption-boundary peers, reasoning was still judged against every peer, so a blob only a rejected peer could decrypt reached the tagged one. Pin and strip against the tag-kept peers. * fix(router): mark the deployment TypeVar Final * fix(router): keep every boundary peer for the router's tag check Returning only the tag-kept peers hid tags the later tag check needs to fail open, so a request that passed the affinity check was refused. Return all boundary peers and judge the reasoning against the kept ones. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
joshua-berri
|
7356ec9f94
|
fix(router): let a retry leave a pinned deployment after a retryable failure (#50769)
* fix(router): let a retry leave a pinned deployment after a retryable failure The retry loop only recorded a failed deployment for non-retryable statuses, so encrypted_content_affinity re-pinned the failing origin on every 5xx, 429 or connection-error retry. Track every deployment the request failed on and hand that list to the callback filters, leaving selection's skip list alone. Fixes #47607 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): carry failed deployments through stream retries too retry_stream started each retry with no failed ids, so an affinity pin on a stream that failed early stayed on the deployment it failed on. Seed the list with the stream's deployment and add each retry that fails. Drop the 429 python case, whose RateLimitError mock cannot be deep-copied by the router. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(router): cargo fmt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): leave out deployments that failed the request when affinity strips When the origin had failed this request and no boundary peer existed, the affinity check stripped the reasoning and returned every candidate, the failed origin included. Later retries then carried no marker and re-picked the failed deployment one time in three. Return the untried candidates instead. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
5d6f32a6f6
|
feat(router): let deployments declare a shared encrypted content boundary (#50750)
* feat(router): let deployments declare a shared encrypted content boundary Deployments that set the same model_info.encrypted_content_boundary are treated as encryption boundary peers by encrypted_content_affinity, so regional endpoints of one provider organization with different api_base values keep encrypted reasoning when a follow-up moves between them. Deployments without the key keep the api_base and api_key match. * refactor(router): type the encryption boundary peer lookup's deployments * test(router): cover order fallback hops onto a declared encryption boundary peer * chore(ui): regenerate API types for encrypted_content_boundary --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yujonglee
|
574fcea691
|
fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473)
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores RedisCache and the native Redis client each kept their own breaker, so once the cache learned the coordination Redis was wedged, a spend-counter write through the native store still paid a full socket timeout. Every Redis target now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected into both the RedisCache guard and the native client. The Python FSM is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(rust): cargo fmt the shared breaker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(redis): close the breaker when the recovery probe gets a Redis reply error A command error such as WRONGTYPE means Redis answered, but the probe was left unsettled and its drop reopened the breaker for another recovery timeout. Both the RedisCache guard and the native client now settle such a probe as a success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw. * fix(redis): key the shared breaker by server and close the three audit gaps The breaker registry now keys on the Redis server (host and port, unix socket, cluster node set or sentinel service) instead of the full connection target, so caches that differ only in db, credentials or url spelling share one breaker. A READONLY reply from a replica mid-failover no longer counts as Redis answering, so it can't reset a run of failures. A new RedisCache refreshes the breaker state gauge, which the deleted Python breaker did at construction. A shared native client is only reused by a store that passes the same breaker. * refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations lib.rs now only declares modules and re-exports, with resolution, target, tls, auth and endpoint each in their own file. Derived FromPyObject already reads a struct field as the attribute of the same name, so #[pyo3(attribute)] on the breaker and store settings did nothing. * refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot Derived FromPyObject already reads each field as the attribute of the same name. The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in the same order. * refactor(redis): return a HashMap from the three memory-tier snapshots The snapshots read cache-memory's HashMap, so there is no order to keep. PyO3 builds the same dict from a std HashMap, which replaces the hand-written PyDict loops in the spend counter and router usage stores and the IndexMap in the cooldown store. The rate-limit slot registry keeps IndexMap because it hands back a Python dict in the order Python wrote it. * refactor(redis): share Redis breakers and clients through one RedisHosts per host redis-connect's RedisHosts now owns the per-server breaker and the shared clients that python-bridge kept in two process-wide statics. A client takes breaker settings and admits through its server's breaker, so no caller can pair a client with a breaker of its own, and clients are keyed by Resolution alone. This drops RedisClient::guarded_by and Breaker::same. python-bridge keeps one RedisHosts per process and calls it with the GIL released. The gateway builds one per connect, so its coordination and response cache on the same server now share a breaker. RedisStoreSettings.breaker is back to breaker settings, as on main. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
tessellark
|
b5a72c2042
|
fix(router): pin /v1/responses requests for prompt caching affinity (#50321)
* fix(router): pin /v1/responses requests for prompt caching affinity * fix(router): pin previous_response_id turns and keep native tools in the Responses pin key A Responses turn that continues a stored response now goes to the deployment that answered it, unless that deployment already failed this request. Native Responses tools are hashed into the pin key as sent instead of converted, so two requests with different native tools no longer share a pin and a native request no longer logs a false dropped-tool warning. The pin keeps its cacheable deployment ids in a sorted tuple so loggers can serialize request metadata. * fix(router): leave a Responses request with a malformed tool to the provider * fix(router): keep the Responses prompt cache pin out of logged metadata The pre-call check now keeps the pin in a router-owned in-flight store and puts only a random token in litellm_metadata, so loggers that ship metadata never see the routing key or deployment ids. The store is shared across check instances, so turning prompt_caching off and on at runtime keeps pinning, and a retry drops its previous attempt's token. The skipped-input debug log now names only the error type. * fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries The prompt caching check no longer narrows the candidates inside the callback filters. It names the deployment it prefers, and the router core applies that preference at the pinned deployment step, after tag, region, pre-call and plugin filters, only when the deployment is still a candidate. A pin can no longer send a request to a deployment its tags do not allow. A retry skips the pin when the pinned deployment was already tried in this request, read from the retry breadcrumbs in both metadata buckets and the excluded and retry-skipped ids. When the retry count outruns the breadcrumbs the check stands down, so a pinned deployment that keeps failing is not retried on every attempt. * test(router): list the prompt caching check as an in-memory cache caller The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers * fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records * test(router): lift test cooldowns through the cooldown store's evict_local * fix(router): key Responses tools like their chat twin and drop the pin token after success A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict. * test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yujonglee
|
b1b236826a
|
feat(redis): serve the proxy's common Redis call with Rust stores (#50170)
* feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Kerry Lu
|
8ac9da84f7
|
fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools (#50275)
* fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools * test(integration): build the web search filter proxy config without mutating the shared config --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
3f13c1ca34
|
fix(router): write the prompt caching pin when turn_off_message_logging is on (#50276)
* fix(router): write the prompt caching pin when turn_off_message_logging is on The success event PromptCachingDeploymentCheck reads had its messages redacted in place by turn_off_message_logging before any callback ran, so the pin was keyed off redacted-by-litellm, never written, and every follow-up bounced across the group. Callbacks that route on the request and never log it now opt out of that redaction and get the request messages back, while every other callback and the spend log keep the redacted payload. * fix(logging): type the request-messages payload helper as dict[str, object] The bare dict return grew the LIT001 count past the merge base and left the returned fields unchecked * test(integration): type the pin rig's Redis client as Redis[bytes] The untyped generic grew the basedpyright reportUnknownMemberType count past the merge base * test(integration): quote the Redis[bytes] annotation so the rig collects at runtime redis.Redis is not generic at runtime, so the unquoted subscript raised TypeError on import * fix(logging): read the request messages once as object in the exempt payload helper Two partially unknown reads of self.messages grew the basedpyright reportUnknownMemberType count past the merge base --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
e011a4ea9b
|
fix(router): do not cool down a deployment on a 404 from a response id lookup (#50262)
The Rust core records every attempt failure against the routed deployment, so a client GET, DELETE, cancel or input_items call on a response id that the provider answers with 404 counted toward cooldown. The removed Python core never saw these failures. Exempt them in failure_exits next to the background poll 404 and video lookup exits. |
||
|
Nate Armstrong
|
067f93340f
|
refactor(router): delete the python router core, the rust core is the only core (#50178) | ||
|
Nate Armstrong
|
9752869d84
|
feat(router): add the Rust router core with the host as its only boundary (#49513)
Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/), served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in the host-side litellm/router_rust/ package (steps/, invoke/, surface/) driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state in litellm/router_rust/surface/ (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. The crate records each attempt's outcome itself rather than through litellm callbacks. A streamed attempt is deferred: it counts its request when the stream opens, and the host's stream wrappers report the success with the final usage, or a failure after the first chunk, when the stream ends, as the Python core's callbacks do. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores, with nothing pending; only the names of the Python core's own outcome callbacks are asserted on Python alone. litellm.Router is now final: subclassing it raises TypeError at the class statement, and the machinery that kept subclass overrides working is gone. The Rust core never honoured subclass overrides, so they would have silently stopped running there. Custom routing goes through register_plugin(), settings through apply() and outcomes through observe(). This is a breaking change for anyone subclassing Router. Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries, max_fallbacks, allowed_fails, retry_after, timeout, the tag settings, routing_strategy_args and the fallback lists as given (the crate reads them as numbers, update_settings casts as Python does); a deprecated `model_list = [...]` write reaches the Rust core; a failed set_model_list is all or nothing on both cores (the deployments served before the call are put back and the error raised); config_deployments() is a generator and `del` of a plain attribute works on the Rust core. Least-busy pick reservations from main now work on the Rust core: its selectors share one reservation store, routed calls reserve and free their unstarted picks, and select() and batch cancel do not reserve. On the Python core a dropped upsert takes its name back off deployment_names, and get_deployment_order reads every numeric spelling as one int so mixed orders no longer raise TypeError. Rulings on the remaining differences, the Python core changed where it was the odd one out: sync picks and picks by deployment id apply the team filter as async picks do, so another team's deployment is refused; the fallback lists and default_litellm_params in RouterSettings are copies, so an in-place edit is ignored and settings change only through apply(); one health record with an unreadable timestamp drops out alone; an empty fallback entry is skipped; a bool cooldown_time is refused at construction, a bool or string cooldown_time on a deployment or a call falls back to the router default, and a failure status that no HTTP status can be is recorded as 500; an infinite or NaN cost and a whole number wider than 64 bits (ptu_shares) are refused on both cores with the same error (litellm/router_utils/deployment_numbers.py); and a /v1/messages stream whose provider iterator raises cools the deployment and retries like a non-stream call, with the Rust core waiting for that cooldown before its retry picks. Differences the proxy parity fuzz and its regression tests found on the Rust core: a file upload or a batch on an uploaded file falls back only within its own model group, as on the Python core, since the file lives with one provider (the host sends same_group_fallbacks_only and the walk skips targets in other groups); Decisions calls drop deployments whose provider has no Decisions config, and deployments that refuse the audio or videos a call carries, unless that would leave none, as the Python core does, while the retry count still reads the whole group; a file upload that fails on one order level is created on the next level's deployments rather than the first level again; and a response that cannot be converted (a text_completion body without a message) reports only its own traceback, so a rate limit handled by the attempt before it no longer turns the error into a 429, and both cores return 500. |
||
|
zoveki
|
784754ca48
|
fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863)
* fix(azure): answer 401 when Entra rejects the deployment's credential A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500. * fix(azure): keep network failures to Entra a retryable 500 azure-identity wraps a transport failure in ClientAuthenticationError, so the 401 mapping now skips errors whose cause chain holds ServiceRequestError or ServiceResponseError * fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials * fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch maps the same rejection, and the A2A endpoint answers it with HTTP 401. * fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown * fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections ClientSecretCredential drops Entra's HTTP status, so throttling and transient codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable instead of mapping to 401. The A2A endpoint answers 401 only for an AuthenticationError caused by an Entra token rejection, and the router skips cooldown for a caller-sent Entra field only on Azure-family deployments. The azure/ Decisions route and the Bing grounding search tool now map an environment-credential rejection to the same 401. * fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents * fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting The fallback hop skips cooldown when Entra rejected a credential the caller sent, Prometheus counts a deployment's own Entra rejection as a deployment failure again, and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router. * fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message A caller-sent Entra field now skips the deployment's cooldown only when it differs from the deployment, its named credential, the router defaults, the AZURE_* environment variables and the default scope. An empty value counts as unset, so a deployment whose own secret is wrong still cools down when the caller's field changes nothing about the token request. The /azure passthrough's Entra 401 message goes through the same internal hostname redaction as other client-facing errors. * fix(router): compare a caller's Entra field with the value the deployment uses A caller-sent Entra field skipped the cooldown only when it matched no configured source, so a value equal to an environment default the deployment overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own and let a caller's rejected credential cool down a working deployment. The check now resolves the value the deployment would use without the caller, in order from the router defaults, the deployment, its named credential, the AZURE_* environment variables and the default scope, and skips the cooldown only when the caller's value differs from it. * fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged * fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider * fix(azure): keep the streaming Entra rejection inside the failed attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
baddc67041
|
fix(anthropic)!: carry prompt_cache_breakpoint to cache_control on chat completions to Claude (#49783)
* fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude
A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped.
Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it.
* fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap
Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and
keep the earliest under it, counting the client's own message, tool, and
top-level cache_control marks. The ttl and malformed-marker errors name
drop_params for proxy users too and no longer name a single provider.
* fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models
A client tool cache_control or a tool_config injection point becomes a Converse tool
cachePoint, so it now takes one of the 4 slots before a translated marker does.
Converse only translates prompt_cache_breakpoint for a model the cost map says takes
prompt caching. An unmapped model such as an application inference profile ARN opts in
through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a
mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403.
Router prompt-caching affinity now treats a marked block like a cache_control one, and
requests without a marker skip the translation after one pass over their messages.
* fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap
* fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results
* perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast
* fix(anthropic): drop the duplicate drop_params left by the main merge
* fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model
* fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap
Under drop_params the translated marks that fit the cap are now split between
the start and the end of the request, so the stable prefix and the newest turns
both stay cached. The cap count also includes a native tool search tool that
keeps its cache_control on the wire.
Bedrock Converse now translates the marker only for the Claude and Nova
families, which Bedrock documents as taking cache checkpoints, unless the
deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its
cache mark when the request fits the cap.
* refactor(bedrock): decide guarded cache marks with a typed bool helper
* fix(bedrock): validate guarded blocks before reading their cache mark
* fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated
When a deployment sets cache_control_injection_points, the client's
prompt_cache_breakpoint marks are dropped on targets that do not read them, so
the operator's injection points decide every Claude breakpoint.
OpenCode Go models served over the Anthropic messages endpoint translate the
marker only for Claude, and guarded Converse text keeps a cache mark only on a
model known to take cache points.
* fix(anthropic): keep client marks unless a message injection point replaces them on a translating route
A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark.
* fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection
A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect.
* fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params
* fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
|
||
|
tessellark
|
ff91b89b8d
|
fix(router): keep encrypted reasoning when a fallback hops between same-key deployments with no api_base (#49396)
* fix(router): keep encrypted reasoning on a hop between same-key deployments on the provider's default endpoint * test(router): type the default-endpoint boundary test helpers * fix(guardrails): restore the google_text_moderation embeddings fix an earlier main merge dropped * fix(router): match encryption boundaries on the key, account, and region the request really uses A deployment's named credential now fills only the values the deployment leaves out, through the same helper the request path uses, so the boundary names the key and base the request sends. A deployment with no api_key takes the provider's default key, the boundary carries the OpenAI organization and project a deployment sends, and a Bedrock Mantle region named in the model routes through the provider resolver before the Responses URL builder * fix(router): keep header-signed accounts apart on a shared boundary and skip boundary work no reasoning needs Headers the request logger masks as secrets (Authorization, api-key, x-api-key) now join the boundary's account scope, with extra_headers overriding headers the way the request merges them, so deployments that sign in through a header on one base never share encrypted reasoning. Target boundaries are built only when a reasoning item from another deployment needs the comparison, and the Responses input strip asks only about encrypted reasoning items, so a pinned turn or fallback hop with nothing to compare makes no secret-manager reads. * fix(router): resolve default boundaries only for openai and drop the unsent organization param Aggregators and Bedrock Mantle serve several vendors on one key, so a missing key or base there no longer yields a shared boundary. The Responses request never sends the organization param, so it no longer splits the boundary, while a Cookie header now does. * fix(router): match explicit pairs as on main and key default boundaries by model An explicit api_base plus api_key pair is the whole boundary again, so tag headers no longer split deployments that main kept together. The default OpenAI boundary now carries the model, since one OPENAI_BASE_URL can front several vendors, and a deployment that declares another provider gets no boundary before any secret lookup. * fix(router): match an explicit key and the environment key on one api_base A deployment with an explicit api_base and api_key now shares a boundary with one on the same api_base that takes the same key from OPENAI_API_KEY, when the model and account headers also match. Two explicit pairs still match on base and key alone, as on main. * fix(router): skip a deployment whose params cannot be read in boundary checks A named credential with a value of the wrong type, or an extra header whose name is not a string, made the boundary check raise and fail the whole request. That deployment now has no boundary, so the others still keep the reasoning they can decrypt. Two explicit api_base plus api_key pairs also compare without resolving the provider, so they read no secrets, as on main. * fix(router): ask a secret manager for default boundaries at most once a minute per deployment * fix(router): resolve a changed named credential at once in the cached boundary and take an injected clock --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
22d0b1e443
|
fix(router): restore four base behaviors the RoutingContext refactor changed (#49395) | ||
|
Nate Armstrong
|
c43a1addf2
|
refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119)
* refactor(router): move litellm/router.py to litellm/router_python_core.py A pure rename with no content change, so git and review tools track the router body as a move. The next commit turns it into PythonCore and adds the Router facade back at litellm/router.py * refactor(router): extract PythonCore behind RouterCore and pass the router its config The routing logic moves unchanged from litellm/router.py into PythonCore in litellm/router_python_core.py. Router becomes a facade that builds a RouterConfig, picks a RouterCore with make_core and delegates every member to it. The core no longer reads or writes litellm.* directly: process settings come from RouterDynamicConfig, and inference calls, sleeps, callback lists and the global cache go through PythonHost. A CI check bans litellm.* in the router modules, and the proxy's construction sites build the config outside the router Settings are static: the constructor arguments and changeset values are copied when the router is built or the change applied, and a write to a setting attribute becomes apply(ChangeSet(assign=...)). A Router subclass's core class also subclasses the facade, so an override's super(Sub, self) works, and an instance-level patch of a core method is kept on the facade so nested patches unwind. tests/unit/test_router/core tests the core in isolation: ScriptedHost serves every host operation (scripted inference answers, recorded sleeps, settings, callbacks, cache), and make_core runs each test once per core in CORE_BUILDERS |
||
|
Nate Armstrong
|
87d91d78b7
|
refactor(router): route the router family through RoutingContext and close the outside surface (#49118)
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality sub-routers now take a RoutingContext instead of the Router, and read the router only through registry(), settings(), health() and the context's forwarding methods. Router-free helpers move from router_utils to litellm_core_utils, outside callers of router state switch to entry points, and the surface allowlist shrinks to the endpoint methods plus the entry points |
||
|
quietjules
|
d302096b5a
|
feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex (#48949)
* feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex lite pi, lite opencode and lite codex now read supported_reasoning_efforts from /model_group/info and write them into each client's model config. pi gets reasoning and a thinkingLevelMap, OpenCode gets reasoning_options plus one variant per effort, and Codex fallback entries get supported_reasoning_levels with a default level. The proxy-served Codex catalog on /v1/models?client_version= offers the same levels Codex before 0.138.0 fails the whole catalog on an effort it cannot name, so those releases, and an unreadable version, get only the six efforts every release parses. A group with unknown efforts keeps today's output, and a group with none is written as non-reasoning The strength order and nearest_declared_reasoning_effort move to an SDK-free module so the lite CLI, which loads without initializing the SDK, can import the Codex catalog * fix(lite): describe team models by their public name and keep Codex's none default /model_group/info now looks up a team's own deployments when the caller's team serves the requested name, so a team model reports its reasoning levels. Codex fallback entries default to none when the group lists it, and the proxy builds the Codex catalog in a worker thread * fix(lite): round Codex's default level down and describe a team's shared-name model from its own deployments Move the reasoning strength helper to litellm_core_utils so lite and the Codex catalog import it without the SDK or the router, and let make lint-format-check-changed diff the working tree so a staged rename passes. * fix(lite): keep pi's system role on reasoning model groups pi switches its system prompt to the developer role for any model marked reasoning, so an operator's cache_control_injection_points entry for role system stopped matching on Claude groups. lite pi now sets compat.supportsDeveloperRole false on those entries, so pi sends system as it did before. * fix(codex): treat a client_version too long to parse as an old Codex and keep undescribed stock levels A client_version past Python's integer digit limit made the Codex catalog answer 500. It now gets the levels every Codex parses. A stock Codex level with no description validates again, as on base, and lite codex treats undecodable codex --version output as an unknown version * fix(lite): keep REASONING_EFFORT_STRENGTH_ORDER importable from router_utils and skip the Codex sync on undecodable debug models output --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
a659e8be70
|
refactor(router): add router entry points and keep the old members as a deprecated mixin (#48840)
Router gains eight entry points (registry, settings, health, select, apply/aapply, observe, register_plugin, retire) in litellm/router_api.py. The 123 public members outside code used become DeprecatedRouterMembers (litellm/router_deprecated.py): same signatures and results, a DeprecationWarning naming the replacement and a use counter for callers outside the router. Router internals move to _core_ names; subclass overrides of old members are still called by the router. |
||
|
Mateo Wang
|
6cdb9959df
|
feat(router): opt in to non-chat deployment filtering (#46257)
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior. * fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions * fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog * fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags * fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released * fix(proxy): walk fallback layers without recursion and gate the compact Responses route The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes * fix(proxy): keep chat on rows whose cost map entry serves chat A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s * fix(proxy): check chat sent to a deployment id or backend model name * fix(proxy): check a model the read-through loads from the database * fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400 * fix(proxy): ignore a client model_group that is not a string when building rate-limit headers A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells. * fix(proxy): keep a non-string client model_group out of spend rows * fix(proxy): validate standard fallbacks before the typed fallback-group lookup * feat(router): opt in to non-chat deployment filtering * fix(router): skip unresolved local rate-limit fallbacks * fix(router): ignore caller-provided endpoint selector values * fix(router): normalize declared chat endpoint prefixes * test(router): cover preflight error-value behavior * test(router): derive replay metadata defaults from model info * fix(router): preserve fallback aliases and check synchronous text calls * test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission * test(routing): keep the endpoint gate's owned proxies on the worker's database * fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None so an omitted flag keeps the stored value The same-group retry of a stream dropped before content called the provider without the endpoint selection context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside endpoint_selection like every other router attempt * fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow Main's router fence lets code outside the router use only listed Router members and router_utils modules. The proxy's fallback probe now calls the listed get_routable_deployments with the request's call type, and the gate's 400 moves to litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's UnsupportedEndpointError, which the router retries on the group's other deployments, while the gate refuses the whole group * test(responses): cover the WebSocket error frame for a model that refuses the endpoint * fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup * fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame * fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path * test(router): type the load-warning test's caplog and router * test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket --------- Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain> |
||
|
Mateo Wang
|
0cf2bb7296
|
fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761)
* fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params A GPT-5 request whose reasoning_effort the model map does not enable for that model lost the effort with no log line under drop_params. An Azure deployment name missing from the cost map lost reasoning_effort none this way and ran at the model's default effort. The gate now logs a warning naming the model, the level, the map entry it checked, and the model_info.base_model and model_info.reasoning_effort_levels fixes. * fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from its underlying model. * fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops * fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level * fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses * fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop * fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live. * fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels (github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact suggested levels it never reads either. Declared levels are now None when a caller passes none, so the advice appears only when a caller fed them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model. The openai row of the chat warning test now expects no base_model advice, the Responses warning test goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped. * fix: read only the built-in catalog when choosing the reasoning drop warning |
||
|
Nate Armstrong
|
7650f766b0
|
refactor(router): fence the router surface and document the router-core boundary (#48839)
Add rustdoc overviews for a new litellm-router-core crate and for litellm-model-catalog, a module docstring on litellm/router.py, and a shrink-only allowlist of the Router members and router_utils modules used outside the router, checked in CI by router_surface_check.py. Move the router-free helpers out of litellm/router_utils: batch_utils to litellm_core_utils, and the auto-router naming contract, tuning baseline and baseline request next to the auto-router plugin. No behaviour change. |
||
|
Kerry Lu
|
96fe78f96e
|
fix(router): reject string fallback targets at Router init (LIT-10238) (#48943)
* fix(router): reject string fallback targets at Router init (LIT-10238) * fix(router): keep direct target fallback entries when rejecting string targets * test(router): set string fallbacks after init in cooldown tolerance tests --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
84c4c5aa5c
|
fix(router): cool down failing deployments when fallbacks use string targets (LIT-10238) (#48913)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
shrey
|
a45c2023ce
|
fix(audio): return 400 for unsupported transcription providers (#47971) | ||
|
simonp9
|
78d6c1aa82
|
fix(router): pass lifecycle events through on background OpenAI responses streams that cannot fall back (#48817)
* fix(router): pass lifecycle events through on background OpenAI responses streams that cannot fall back The Router held response.created, response.queued and response.in_progress until the first output event on every /v1/responses stream, so a background run showed the client no response id while it was queued or reasoning. A native OpenAI stream whose upstream echoes background: true now releases those events as they arrive when no fallback, order level or weighted failover could replace it, and once the id is out a later pre-output error is raised instead of starting a second run behind it. * test(router): type background stream test params and wait on the upstream pause instead of a clock * fix(router): keep the content-policy note on a released background stream error * fix(responses): return the routed id from cancel and add the router note only before output acancel_responses and cancel_responses now re-encode the reply id with the deployment, like get. The released stream's content-policy note follows the held path, which adds it only before output text. Renames _anthropic_messages_order_levels to _stream_order_levels. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
8e2e99670d
|
fix(router): continue Responses mid-stream fallbacks on models without assistant prefill (#47672)
* fix(router): avoid assistant prefill in Responses fallbacks * fix(router): keep prefill for Claude fallbacks that take it and skip constrained continuations A Responses fallback to a Claude model that takes assistant prefill now gets the partial response as prefill again, so it continues mid-word without the prompt-shape drift. Every other fallback keeps the request to continue. A stream cut after text no longer continues when the request constrains its output (a json_schema or json_object text format, or a forced tool_choice), since a continued JSON document or tool call would not parse. The client gets the primary's error, as on the chat path. The fallback's re-sent leading space at the seam is dropped, and the prefill shaping runs per deployment after the encrypted reasoning strip, so stripped reasoning is never replayed. * fix(router): guard SDK text_format, narrow forced tool_choice, read Claude base_model, and cap Responses continuation output Cut Responses streams that asked for structured output through the SDK's text_format now surface the primary's error instead of stitching two models' JSON. Only a required or named tool_choice counts as forced, so allowed_tools in auto mode and hosted tool choices still continue. A Claude deployment named only through base_model keeps its prefill continuation, and the continuation's max_output_tokens is lowered by what the cut stream already emitted. * fix(router): quote the partial's ending in the continue request and surface the primary's error after a cut tool call * fix(router): stop Responses fallbacks from continuing content-filter cuts and client-run tool calls A content filter that cuts a /v1/responses stream mid-answer now returns its error, as on chat, instead of letting a content_policy_fallbacks backup finish the filtered text. Shell, apply_patch, computer, local_shell, and MCP approval items count as a started tool call. Chat continuation goes back to treating any tool_choice object as constraining, as on main * fix(responses): count a stream of tool arguments as a started tool call * fix(router): let Responses fallbacks continue under allowed_tools auto and after server-run tool calls A Responses tool_choice of allowed_tools in auto mode no longer blocks the continuation, since the model may still answer in text. Hosted MCP calls and the bridge's server-run tool search no longer count as a started client tool call. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Aidan Singh
|
b97fcf1af0
|
fix(router): let affinity pins yield to deployments that already failed this request (#32309)
* fix(router): apply weighted-failover exclusion before deployment-affinity callback * fix(router): let affinity pins yield to deployments that already failed this request Moving the weighted failover exclusion ahead of the affinity check also changed what tag, order, plugin and managed-file filters saw. Restore the original filter order and have the affinity check ignore session and API key pins that point at a deployment already excluded or retry-skipped in this request. previous_response_id pins still win. * fix(router): release affinity pins on deployments that failed this request Delete a session or API key pin whose deployment already failed the request, so the deployment that serves it claims the pin. Prompt caching pins skip failed deployments too, and the proxy strips the internal failed-id fields from client bodies. * fix(router): keep affinity pins in place and skip failed origins in encrypted content affinity A pin now only yields for the request whose pinned deployment failed. Deleting it moved pins after one 429 or a request-caused 400 and added a Redis call on the routing path. Encrypted content affinity also ignores deployments that failed this request, so a failed origin takes the strip-and-dispatch path. * fix(router): keep encrypted reasoning when a retry can only go back to its origin The encrypted content check now drops failed deployments the way the router does: a weighted failover exclusion always holds, and a retry skip that would leave nothing falls back to the whole group. The proxy also drops client-sent metadata._failover_excluded_ids, and the encrypted content tests use a real Router * fix(router): keep encrypted reasoning when previous_response_id binds the failed origin --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
ff65785717
|
fix(router)!: count an explicit allowed_fails of 3 and keep fallback deployments out of cooldown (#46183)
* fix(router): count an explicit allowed_fails equal to the default as set Router(allowed_fails=3) stored the same value as an unset Router, so _is_allowed_fails_set_on_router treated it as "not set" and kept the rate-based cooldown logic. Keep the passed value in explicit_allowed_fails, like _explicit_timeout, and count it as set. * chore(router): drop the comment on explicit_allowed_fails * fix(router): keep the fallback deployment out of cooldown when no deployment can be picked When every deployment of the requested model is in cooldown, the router raises "No deployments available" and schedules the failure callbacks on the request's logging object. By the time they run, the fallback has already written its own deployment into that object, so the 429 counted as a failure of the healthy fallback deployment and cooled it down too, leaving the model group with nothing to serve. Tag the selection error and skip cooldown accounting for it, the same way advisor orchestration failures are skipped. * test(router): annotate the new cooldown test variables with Final * test(router): pin that a saved allowed_fails of 3 stays unset and type the fallback helper * test(router): use a fixed start time in the fallback helper * test(router): cover a fallback group staying up after the primary's selection fails --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
8f31c7171d
|
fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception (#48644)
* fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): give cooldown fixtures a sibling deployment so allowed_fails=0 still forces a cooldown * fix(router): keep benching a lone deployment that has a fallback group or an empty deployment policy * fix(router): accept dictionary fallback targets in the lone deployment check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
14193435b8
|
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
7e14638001
|
refactor(llms): rename model-version transformation files to family names (#48509)
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure), the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py, anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py, and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol names are unchanged. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
55479821e2
|
feat(router): add opt-in session repinning after successful failover (#48206)
* feat(router): add opt-in session repinning after successful failover * fix(router): respect shared session pins and split metadata * fix(router): preserve session routing during Redis failures * test(router): inject the delayed Redis reader |
||
|
Mateo Wang
|
820725661e
|
fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage (#45194)
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception, and a second drop never reached the failure handlers because the first had already marked the object as logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the router discards the stashed usage right before it opens the next attempt, on same-group retries and on every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage. * fix(router): dispatch chat stream failures through the attempt-scoped logging copy The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types. * test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy * test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead. * test(router): use real Logging in fallback cooldown test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve superseded retry usage for billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bill retry input after prior stream usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): track retry dispatch for terminal billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): use deterministic retry timestamp Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): avoid real clock in fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): remove duplicate streaming logger import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retain error-frame usage for exhausted retries * test(proxy): fix retry billing expectations and helper annotations * fix(proxy): preserve failure estimate without carried usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): audit retry helpers and fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): fix retry fixture timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): observe retry stash at attempt boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): keep terminal failure attribution when dropped attempts log on copies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish a dropped attempt's payload only while it is still the current attempt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover spend-log encoding error propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): cool down the final fallback attempt when its failure callbacks were blocked Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): import Final in adapter regression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop spend-log encoding expectation superseded by main's poison isolation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): finish the wrapped stream's failure log before surfacing the error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): pin pass-through stream fixture start times Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload before awaiting failure callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload from the failure handler after baseline invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging): drop unused discard_partial_usage_for_failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
0c6731eb5c
|
fix(router): price cost-based routing through model_info.base_model (#47617)
* fix(router): price cost-based routing through model_info.base_model * fix(router): keep cost-based routing pricing helpers fully typed * fix(router): drop typing.cast from cost-based routing pricing helpers * fix(router): read the cost map through a typed accessor in cost-based routing * fix(router): avoid a Final rebind inside the cost-based routing loop * refactor(router): share deployment_pricing_models across spend, listing and cost-based routing * refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing * refactor(router): inline the deployment pricing-model wrapper in cost-based routing * refactor(router): read deployment pricing models through one shared helper deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model / litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model) stays for the failure-row spend estimate, which already holds the two names * refactor(router): share the str-keyed mapping validator with cost-based routing * ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
6564a490f4
|
fix(batches): retry confirmed pre-accounting failures (#47625)
* fix(batches): retry confirmed pre-accounting failures * fix(batches): avoid promising retries for unknown accounting outcomes * test(batches): exercise real spend writer during retry recovery * fix(budgets): charge retried batches once per router budget * test(budgets): cover shared batch charge helpers --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
c38b86cc93
|
fix(router): skip non-dict tool entries in the prompt caching affinity key instead of raising (#47565)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Vincent Gimenes
|
808333682e
|
feat(embeddings): enforce per-request input batch limit (#44574)
* feat(embeddings): add max_batch_inputs pre-call validation Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> * add unit tests for max embedding batch size Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> * feat(embeddings): enforce batch limit on extra_body input override Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> * chore(dashboard): update schema.d.ts for embedding batch size model field Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> * fix(embeddings): close Bedrock batch bypass and satisfy ruff PT012 Count max(top-level input, extra_body.input) for batch limits so providers that ignore extra_body cannot bypass max_embedding_batch_size. Add router tests and refactor pytest.raises blocks for ruff-tests. Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> * fix(embeddings): drop unchecked casts in router batch-size check Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> * fix(embeddings): route oversized batches to a group deployment that admits them * fix(embeddings): count extra_body input for any top-level input and prefer admitting deployments before cooldown * fix(embeddings): log cap rejections to failure callbacks and scope batch routing to embedding calls * fix(embeddings): apply the batch cap after tag and pre-call filters and fall back on cap rejections * fix(router): keep embedding cap rejections out of deployment health and hold only for routable cooling deployments * fix(router): clear the embedding cap rejection mark before a fallback's provider call A fallback or retry shares the request's logging object, so the mark a rejected attempt set made telemetry skip the fallback deployment's real call. Unit tests now pass a fixed logging start time instead of reading the clock. * fix(router): log embedding batch rejections once, before a fallback runs * refactor(router): drop docstrings that restate the code --------- Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d698106ed7
|
fix(router)!: schedule default_priority requests once and reject a malformed priority (#47498)
* fix(router): schedule default_priority requests once and reject a non-integer priority A router configured with default_priority failed every request that carried no priority with a 500 naming a missing 'priority' argument, and an explicit priority re-entered acompletion and hit the same error. acompletion now resolves the priority once (the request's integer, else the default, with 0 kept as a real priority) and schedules through _schedule_factory directly, while schedule_acompletion delegates to acompletion so the two paths share one queue pass. A non-integer priority answers 400 naming the param, or is dropped under drop_params, instead of leaking to the provider or crashing the scheduler. * fix(router): honor router-level drop_params and schedule prompt-management models under a priority * fix(router): drop a non-integer priority when the model's deployments set drop_params * fix(router): resolve the drop_params deployment scope for team routes and deployment ids * fix(router): read the request and router drop_params flags without a loop-assigned Final * refactor(router): type the prompt-management wrapper's return and mock the provider in its test * perf(router): consult drop_params for the priority only when it is not an integer * test(integration): assert the new priority contract for non-integer and zero priorities * docs(router): describe default_priority as the priority every request without one takes * fix(router): parse default_priority once and apply it to text completions and the queue route A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig. * fix(router): type the priority dispatch so the lint budgets hold * fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order * fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses * fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority A request whose model is a deployment id now counts that deployment as healthy in the scheduler, since routing by id already ignores cooldown. Before, the scheduler found no healthy deployment for an id, so the request waited behind every entry ahead of it. A boolean default_priority (true, or YAML off) is ignored at startup with a warning, like any other value that is not a whole number. The scheduler integration tests now share worker state for the defaulted pair, cool a group through /v1/responses so no stale in-memory queue is left behind, and use "high" instead of "1" for the non-integer cases. The concurrent burst test no longer asserts that the queue drains, matching the explicit-priority burst test, since the scheduler's read-modify-write queue can race across workers. * test(router): keep the queue drain check as a BUG-skipped integration test * fix(router): queue default_priority chat requests once instead of failing them * test(router): open with the default_priority regression tests alone * fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name |
||
|
Mateo Wang
|
96f1bc02d8
|
feat(router)!: apply each deployment's reasoning_effort_levels on every provider and route (#46601)
* feat(router): apply each deployment's reasoning_effort_levels on every provider and route A deployment's declared reasoning_effort_levels now decide the effort sent upstream on /v1/chat/completions, /v1/messages and /v1/responses for every provider. Requests round up by default, litellm.reasoning_effort_rounding and model_info.reasoning_effort_rounding switch to down, a declaration lets providers like azure_ai accept reasoning_effort, and each remap logs one INFO line with structured fields. * fix(router): round reasoning effort only on a deployment's own declared levels Cost map levels no longer round requests or force-allow reasoning_effort, so undeclared sibling deployments and catalog-only models behave as on main. A declaring deployment lets azure_ai accept any level the resolver sends, and a hosted_vllm chat thinking budget is rounded too. * fix(router): drop undeclared reasoning effort under drop_params and round native /v1/messages effort Under drop_params a level outside the deployment's reasoning_effort_levels is dropped instead of sent. Providers that take /v1/messages natively now round output_config.effort onto the deployment's declaration. The rounding setting ignores case and surrounding spaces. * fix(router): strip line breaks from every field in the reasoning effort remap log * fix(router): round thinking-budget efforts onto a declaring deployment on every provider * fix(router): keep the caller's thinking display when rounding a thinking budget on adaptive Claude Warn on a reasoning_effort_rounding value other than up or down instead of silently rounding up * test(integration): cover deployment reasoning_effort_levels rounding on chat, messages and responses Wire cells assert the exact body each provider receives when a deployment declares reasoning_effort_levels, plus undeclared controls, malformed input, drop_params, rounding settings, fallbacks, model groups, live updates and the response cache. Chaos cells cover a killed worker, a proxy restart, an upstream outage, a slow stream and the INFO remap log. * test(integration): cut held streams when the upstream stops in the reasoning effort chaos cell The upstream-stop cell waited for the held streams before stopping the upstream, so they only ended on a client or proxy timeout and the result depended on which fired first. The upstream now stops while the streams are held, and each cut stream must carry an error and no answer * fix(router): round thinking budgets only on declaring deployments and round Converse output_config An undeclared hosted_vllm deployment serving a Claude model name now maps a thinking budget the same as any other undeclared hosted_vllm deployment. A model_info that echoes a cost map key no longer borrows that entry's levels. A non-numeric thinking.budget_tokens answers 400, or is dropped under drop_params, where main answered 500. A declaring Bedrock Converse Claude deployment now rounds output_config.effort, which /v1/messages sends for Claude Code * test(integration): cover output_config rounding and non-numeric thinking budgets in the reasoning effort cells * test(integration): keep an explicit reasoning_effort winning over a malformed thinking budget on hosted_vllm * test(integration): cover output_config rounding on openai, azure and azure_ai chat * fix(hosted_vllm): let an explicit effort win over a malformed budget and reject one on /v1/messages An explicit reasoning_effort beside a non-numeric thinking.budget_tokens keeps main's behavior again. A declared hosted_vllm deployment with a Claude model name maps a budget through the generic table instead of the adaptive Claude one. /v1/messages to a non-Claude target hands a malformed budget to completion(), which answers 400 or drops it under drop_params, where it answered 500 * fix(router): drop thinking beside an explicit effort when a declaring deployment's provider cannot read it * test(integration): cover malformed thinking budgets on /v1/messages to non-Claude targets * fix(anthropic): answer 400 for a malformed thinking budget on /v1/messages to any non-Claude model, or drop it under drop_params * test(integration): cover an explicit effort beside thinking on azure_ai and drop_params on the /v1/messages bridges * fix(anthropic): answer 400 for a malformed thinking budget through adapter_completion without drop_params * fix(anthropic): type the adapter_completion request so the drop_params read adds no unknown argument * fix(router): a default effort beside a thinking budget sends no effort on chat Chat now leaves the budget-derived effort out when the caller's effort resolves to none or asks output_config for one, matching /v1/messages. The chat thinking decision moves into resolve_chat_reasoning so completion() stays under basedpyright's complexity limit. * test: type respx_mock and mark the adapter result Final in the reasoning effort tests * fix(router): keep the caller's thinking beside a kept effort and the budget effort beside a dropped one A non-default output_config.effort beside a thinking budget now leaves the thinking as sent, so Bedrock Invoke keeps display updates. A level drop_params drops counts as never sent, so the budget still sets the effort. output_config.effort replaces a thinking budget the provider cannot read, and a non-numeric string budget on adaptive Claude answers 400 in place of 500, or drops under drop_params. * fix(router): send a kept output_config effort as reasoning_effort where the provider cannot read thinking On a declaring hosted_vllm or azure_ai deployment, a thinking budget goes out as reasoning_effort. An output_config.effort beside it now takes that slot, rounded onto the declared levels, instead of leaving only output_config, which those backends ignore. * fix(reasoning-effort): honor global drop_params for a non-numeric budget and keep the budget effort beside a dropped /v1/messages level * fix(reasoning-effort): send a level a gpt-5 deployment declares even where the cost map does not list it * fix(reasoning): apply declared levels on azure_ai gpt-5 and reject a bool thinking budget on adaptive Claude * fix(router): drop a thinking budget's effort that a deployment declaring no ranked levels excludes A deployment declaring [] or ["none"] skipped the budget check, so adaptive Claude still sent the budget-derived output_config.effort under drop_params on chat, Bedrock Converse, and native /v1/messages. The adaptive thinking now goes out without that effort * fix(anthropic): reject a false, null or empty thinking budget on adaptive Claude |