159 commits

Author	SHA1	Message	Date
yujonglee	9a1acee4a7	feat(redis): serve the router's session pins from a native store (#50870) ... Deployment affinity and the complexity router's session and tier pins move off the router's DualCache onto RouterSessionPinStore, backed by the native redis-store. The read and claim Lua scripts run byte-identical to main, the pin read rides the request's pre-call batch, and keys and values keep main's format.	2026-11-13 03:22:28 +00:00
yujonglee	b88e4bff6b	refactor(router): count the whole router_usage family on the native store (#50862) ... * refactor(router): count the whole router_usage family on the native store usage-based-routing-v2, the model rate limit check (tpm, rpm, tpd, rpd) and the io-token check (itpm, otpm) count through the native router_usage store, next to the router core's global_router tpm and rpm. The pick's usage read rides the native pre-call batch and settles on the batch reply, keeping DualCache's per-key read throttle, and the success tpm increment rides the native post-call batch after counting in memory. io-token reservations stash their counter's fields instead of a Redis key. Redis keys and values are unchanged. A Redis service event is reported in a context named for whose it is, either the calling store op's, the request whose post-call batch its deadline sends, or nobody's for breaker transitions and unscoped calls. The deadline's tag is built when a post-call declaration arms it, so that flush parents under the request's span even though auth sets the span after the batch scope opens. An SDK call anchors the span its caller passed, so its usage reads and io-token refunds parent under it without a span argument threaded through the family. * fix(redis): keep an int counter past i64 above the limit instead of wrapping Number::add summed two ints with plain i64 addition. With Redis off and the OTPM counter at 1, a request with max_tokens=2**63-1 passed the MAX_INCREMENT check, and the memory count wrapped negative in a release build, so it passed admission and let other requests past the cap. Python's int never overflows, so main sees a count above the limit, rolls it back and answers 429 An int sum past i64 now becomes the nearest float, which compares above any limit the way Python's int does. The rollback after such a refusal can leave the count one token low, since the float cannot hold 2**63 exactly * fix(logging): run every callback under the span an SDK caller passed Native Redis service spans take their parent from the request span contextvar. The proxy sets it at auth and the SDK router scopes it around the routed call, but a streamed response is iterated after that scope closes, so its callbacks saw no span. Three callbacks reopened the scope from kwargs by hand, and the sync stream path had no wrap at all because its success handler runs on an executor thread The four Logging handlers now anchor the span from model_call_details around their dispatch, so every CustomLogger callback sees it, streamed or not, sync or async, with or without a Router. The per-callback wraps in lowest_tpm_rpm_v2 and model_rate_limit_check are gone. A span already anchored, as on the proxy, is kept * test(logging): cover a mid-stream failure, and drop the span wrap in Router._release The callback span test now also breaks a stream after its first chunk, sync and async, through a custom provider, so the failure handlers the stream wrapper starts on a bare thread are covered too A probe through the Router showed Router._release only runs inside the routed attempt, where the router has already anchored the span, and never after a stream returns. Its own wrap was a no-op, so it is gone * fix(router): refuse an io-token reservation past its limit before counting it The overflow fix let a reservation near i64::MAX become a float, so the count still read above the limit and the request got its 429. But the rollback subtracted the same amount from that float, and with OTPM usage at 40 the count came back as 0.0, so later requests reused capacity already spent A reservation above its limit now gets the 429 before the store is touched, in the async and sync paths. On main the count went up and straight back down for such a reservation, so the answer and the count after it are the same as main's. Only the usage named in the error message differs, which now shows the reservation instead of the count plus the reservation * test(redis): send the failed SDK call's refund through a real completion The refund span test called ModelRateLimitingCheck.async_log_failure_event directly, so it relied on the span wrap that callback used to carry. The Logging failure handlers now anchor the span for every callback, so the test sends a failing litellm.acompletion with the reservation in its metadata and checks the refund's Redis span still lands under the caller's span	2026-11-13 03:22:28 +00:00
yujonglee	e20f07d178	feat(redis): serve the health checks' deployment states from a native store (#50357) ... * feat(redis): serve the health checks' deployment states from a native store DeploymentHealthCache reads and writes its one Redis entry through a new deployment_health store family instead of the router's DualCache, like the other router families. * refactor(redis): type the deployment health states in the store The health store kept a raw Value from Redis to Python and back, so Python re-checked the shape by hand and router-core only got a typed record after a Python pass. The family now has a types.rs with HealthState and HealthStates, the bridge converts through PyData, and the Python store returns DeploymentHealthStateValue mappings Reads keep main's leniency for records another writer left: extra fields are ignored, a falsy flag is unhealthy and a missing one healthy, a timestamp that is no number reads as 0, and a record that is no dict is dropped alone. A rewrite writes only is_healthy, timestamp and reason	2026-11-13 03:22:28 +00:00
Nate Armstrong	07fe19bb15	fix(router): validate request fields and ignore scalar fallback overrides (#51502) ... * fix(router): coerce request fallback overrides and reject invalid values Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): validate request fields and ignore scalar fallback overrides Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): centralize fallback filtering in request model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: nate <nate@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 01:24:23 +00:00
kenneth-berri	be623b5bde	fix(token-counter): count Anthropic server tool history (#51334) ... * fix(token-counter): count Anthropic server tool history Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token-counter): bound nested results and update proxy coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token-counter): include readable web search snippets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 09:35:36 -08:00
Yassin Kortam	db6dfb6bb6	chore: move workspace packages under litellm (#51000) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 04:19:57 +00:00
Nate Armstrong	74d76fd758	perf(router): list a group's deployments once per registry change on the acompletion path (#50749) ... * perf(router): list a group's deployments once per registry change on the acompletion path * perf(router): drop the new docstrings * perf(router): keep the counter cache's mutable-ok on its annotation * perf(router): define the counter cache's alias in the registry	2026-11-11 16:58:52 -08:00
zoveki	22252073e7	feat(bedrock): search knowledge bases by image and return image hits (#51063) ... * feat(bedrock): search knowledge bases by image and return image hits vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits * fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery Text searches on a multimodal Bedrock knowledge base return the same text hits as before, so chat, RAG and logging payloads stay unchanged. Image hits come back only when the caller sent query_image A caller retrievalQuery now answers 400 pointing at query_image, or is dropped under drop_params, instead of replacing the built query unchecked A file_search chat tool that carries query_image or rewrite_query searches again instead of failing, and a non-string query next to an image answers 400 instead of 500. query_image is keyword-only on search and asearch * fix(vector-stores): redact query images and keep store search params out of image checks * fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches * fix(vector_stores): keep a caller's image query through store and router params A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off * fix(cost): leave image search responses out of the cost error text * fix(vector_stores): accept line-wrapped base64 in query_image data Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent * fix(vector_stores): drop rewrite_query false from the chat file_search tool False asks for the default search every store already runs, and MongoDB rejects the field outright, so forwarding it made a MongoDB chat lose the store's context for a no-op flag. True is still forwarded. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-12 00:19:34 +00:00
kenneth-berri	b4227652c3	fix(health): log Test Connection spend with the mode's call type and model id (#50987) ... * fix(health): price connection tests using their resolved mode Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): avoid deployment pricing on alternate probes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): preserve pricing for equivalent model names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): avoid authentication when comparing health check models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover health probe pricing without a deployment model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): pin the clock in the rpm rate limit test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-11 16:07:33 -08:00
Nate Armstrong	59538ff1d9	feat(messages): serve anthropic messages on rust only (#51119) ... * feat(messages): apply modify_params rewrites on rust * feat(messages): mint anthropic workload identity tokens on rust * feat(messages): apply prompt caching injection on rust * feat(messages): serve the easy python-only detectors on rust * feat(messages): retry rejected thinking signatures on rust * feat(messages): serve response cache reads on rust * feat(messages): retry rejected strict tools on rust * refactor(messages): delete the python path for anthropic messages Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a detector that raises) gets UnsupportedParamsError naming the feature, raised from one place in litellm/messages/dispatch.py. Other providers keep the Python fallback. A call whose arguments the handler signature rejects still gets Python's TypeError. Deleted Python code that only served anthropic Messages: - workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses were never eligible; the Rust host mints the token itself) - the anthropic Foundry beta-row and affinity-base branch in async_anthropic_messages_handler, and the "forward" mode of the beta filter that only that branch used The proxy MCP tools detector now uses the light tool precheck so a function tool never imports the gateway handler, since a raising detector now refuses. * feat(messages): forward message threads on rust The Rust path for anthropic /v1/messages dropped the thread body field, so a create or continue turn reached Anthropic without its thread or the message-threads-2026-08-12 beta. It now matches the Python handler from #50976: - the host forwards thread with the other body params - a non-null thread adds the message-threads beta - a well-formed continue turn keeps advisor blocks in history and derives no compaction, per-turn-control or tool-change beta from its messages - the recorded httpx_response carries the request that was sent, for JSON and streamed calls, so the proxy's thread ownership hook resends the create turn's betas on a continue as it does for Python Upstream error.details already reached the exception message unchanged, so thread_not_found relays and keeps the deployment out of cooldown on both paths. Parity cases cover create, continue, streaming, sync, retries, thread_not_found and the ownership hook round trip. * feat(messages): serve mock responses on rust * feat(messages): send caller clients and unsupported tls through python on rust * feat(messages): run pre-request hooks on rust * feat(messages): run agentic loop hooks on rust * feat(messages): run the safeguards classifier on rust * feat(messages): run registered interceptors on rust * fix(messages): match python stream cache gate, logging and interceptor order on rust Store a streamed Messages response in the cache only when Python would: a message_stop event line, no provider error frame and at least one frame with output. Leave dict results untouched by mark_rust_response so hidden params never reach result dicts, logging objects or hook digests. Ask interceptors whether they handle the call after the cache lookup, not at decode time, so a cache hit never calls can_handle. Emit the track_llm_api_timing service event for the provider exchange. Bill and log a stream superseded by a router fallback as the failure of its error frame, with the partial usage warning. Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the autorouter baseline cache sees retried requests. * fix(messages): forward python-only json values to anthropic as python does Accept any max_tokens, sampling scalar and messages collection that Python main accepts, and carry values serde_json cannot hold (ints beyond 64 bits, NaN and infinities, strings with lone surrogates) as validated raw JSON leaves written back verbatim on the wire, so the upstream body matches Python's json.dumps * fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust * fix(messages): log python's plain provider header names on rust streams * fix(messages): match python's request params, logged params and validation on rust Read a non-bool stream by Python truthiness and send it as given, log the caller's stream flag as Python's handler does, raise the raw Python errors main raises for malformed messages, tools, model_info and hook-set litellm_params or providers, read the tool custom-field drop from drops_unsupported_params, log the caller's nested fields that additional_drop_params strips and the Python values of raw JSON leaves, keep a missing stop_reason absent, and map provider-config rejections (missing max_tokens, unknown reasoning effort) as Python's HTTP handler maps them, with Python's text, effort order, model and provider * fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust * fix(messages): match python on hooked streams, empty cache writes and mcp cache hits * fix(messages): join already buffered stream reads and trim the env api key as python does on rust * fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust A stream cut by a transport error now hands the caller the bytes already buffered before raising, as aiohttp does for a caller that reads them before the event loop sees the cut, so the frames and their usage reach the caller and the failure log. Provider replies keep integers wider than 64 bits exactly and read numbers that overflow a float as infinity, through the response, the response cache and the logged result. The provider client follows up to 20 redirects as httpx does, and running out raises Python's text. A provider_specific_header value the ascii codec cannot encode fails before the send like a caller extra header. The deployment success hook gets the stream as AnthropicMessagesStreamingResponse, as Python's handler hands it. * feat(messages): serve proxy mcp tools on rust * test(messages): cover hook, agentic and interceptor edge cases in the parity suite * fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust * fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust A header value Python's client cannot encode as ASCII now fails before the send whichever header carries it, so a caller api_key with a non-ASCII character raises Python's InternalServerError instead of a raw UnicodeEncodeError after the reply. A caller api_base without an http(s) scheme, or with whitespace, control or non-ASCII characters, is sent through Python's client at the URL Python builds, so its relative-URL and invalid-URL errors match, and a base urlsplit cannot split raises Python's bare ValueError before the send. A provider_specific_header of the wrong shape is scoped through Python's ProviderSpecificHeaderUtils first, so a plain string raises Python's AttributeError. * fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust temperature and top_p now deserialize through recognized::exact_float, which keeps an integer where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent upstream and logged as the integer python forwards, and comparisons still go through scalar.rs cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python TypeError marked as a late rejection, and map_failure hands it to the caller unmapped * test(messages): format the non-string tool_result parity helper * fix(messages): flag and log the call before python's pre-send raises on rust The bridge now hands settings() and map_failure() whether the call is async. A provider_specific_header of the wrong shape sets the aanthropic_messages flag before re-raising, as Python's handler has already flagged the call. A base urlsplit cannot split also logs the resolved model, provider and optional params first, as Python raises it only after its logger is updated. The TypeError raised while capping the thinking budget for a string max_tokens sets the flag too * feat(messages): serve the web search short-circuit override on rust through the intercept host call * fix(messages): give the web search entry kwargs its mutable-ok reason * test(messages): drop the body hook gap the rust short-circuit closes * fix(messages): name the partially unknown handler types on the short-circuit host call	2026-11-12 00:07:20 +00:00
joshua-berri	8f42db985d	fix(router): map Gemini stream-opening errors (#51218) ... * fix(router): map errors when opening Gemini streams * fix(streaming): preserve stream-open exception causes * fix(streaming): limit open-error mapping to provider and transport errors --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-11-11 12:18:44 -08:00
devin-ai-integration[bot]	3a91fdfc07	fix(management): keep classifier deployments serving during reloads (#51111) ... * fix(management): track self-heal for test_moving_the_workspace_under_a_burst_moves_the_classifier_and_logs_every_request_once Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(management): keep classifier deployments serving during reloads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-11 18:48:06 +00:00
mika-lt	373fe799e7	fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order (#50971) ... * fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order A purpose=batch upload on a mixed Azure chat and batch group was order-filtered before batch routing narrowed it, so the file landed only on the order-1 chat deployment and the batch create that followed got Azure's 400 invalid_deployment_type. The upload now runs under the same batch routing as the batch create. * fix(router): keep a failed batch upload's cooldown and order fallback on the batch deployments * fix(router): keep a batch upload off chat deployments while its batch deployment cools down * fix(router): keep a refused batch upload's retry off the chat deployment * fix(router): keep a refused batch upload's error and skip chat while the batch deployment is paused * test(router): pin the order fallback for a batch create on a group with no batch deployment * test(integration): expect an Azure batch upload on the batch deployment alone --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 18:35:19 -08:00
yujonglee	11af2eb22d	refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) ... * refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 01:57:54 +00:00
Kerry Lu	e2625d0509	fix(router): re-check and reserve per-model rate limits on fallback targets (#50994) ... * fix(router): re-check and reserve per-model rate limits on fallback targets A provider error fallback handed the caller the fallback model group without its per-model TPM/RPM limits ever being checked. The router gets a fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires it to the v3 limiter, which reserves the target model's per-model descriptors before each fallback hop and skips targets that are over their limit. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): dedupe a repeated fallback target before the reserving rate limit check Both walks now consult the attempted set before the rate limit check, so a chain like ["B", "B"] reserves B once. The deprecated settings parity test also learns the new setting. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): a bidirectional fallback chain never re-reserves the requested model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): match the surfaced failure in the fallback reservation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the admission stash untouched while building fallback descriptors Building the fallback target's descriptors could rewrite the request stash's PTU ceiling, which post-call settlement reads for the model the caller named. The fallback check now hides the stash while it builds, like the frame check. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 16:36:10 -08:00
joshua-berri	8fb2b97d5b	fix(router): sync responses() applies the Router's num_retries and fallbacks (#51011) ... * fix(router): sync responses() applies the Router's num_retries and fallbacks Sync Router.responses() made one attempt and raised, while aresponses() retried and fell back on the same Router. The sync plan now walks retries and fallbacks like the async one, stamps the deployment's num_retries on the error the same way, and the SDK no longer nests its own retries under the Router's for sync generic calls Fixes #49938 * fix(router): sync generate_content() retries and falls back through the Router Reading litellm_metadata in the SDK retry check left sync generate_content with no retries at all, since its Router plan made one attempt and the inner completion() call stopped retrying. Sync generate_content and generate_content_stream now walk the Router's retries and fallbacks like their async versions, so retries are neither lost nor nested --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-11-10 12:55:49 -08:00
mrinal-berri	9ba551b871	feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571) ... * feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes	2026-11-10 11:06:17 -08:00
Nate Armstrong	461d3407bf	fix(router): hold the registry lock across router-core writes and mirror reads (#50933) ... * fix(router): hold the registry lock across router-core writes and the mirror reads at their positions A registry lookup returns positions in router-core's list and the host reads its mirror at them, while an upsert wrote router-core and read the mirror back unlocked. A read on another thread in between got an IndexError past the mirror, or after a removal the neighbouring model's entries * test(router): cover the registry lock's read and write paths Move the concurrent-read regression into the registry writes characterization, drop the explanatory lock comments and the unused litellm-model lookup	2026-11-10 17:58:46 +00:00
norkelbyte	68e78bc53a	fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted (#50730) ... * fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted * fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full. * fix(responses): finish a turn's early token count when the client leaves A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors * fix(router): count one router request per Responses WebSocket turn A socket whose finished turns each counted their own request in router usage no longer adds one more request when it closes, so N turns count N requests, as N HTTP calls do. A socket with no counted turn still counts 1 * fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 04:13:51 +00:00
ryan-crabbe-berri	16fad64e95	fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments (#50743) ... * fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments Codex builds separate catalogs for Bedrock's runtime and Mantle endpoints that turn off what Bedrock does not support yet. A model whose deployments run on Bedrock now takes its entry from those catalogs, vendored from Codex 0.166.0, instead of from Codex's OpenAI catalog. * fix(proxy): keep Bedrock catalog entries from Codex releases that cannot decode them codex debug models leaves out fields such as supports_parallel_tool_calls that Codex releases before 0.148.0 require, so a Bedrock entry now needs at least 0.148.0 as well as its source entry's minimum.	2026-11-09 19:00:22 -08:00
yujonglee	523b3efea2	refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) ... * refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:37:35 +00:00
Nate Armstrong	7ed28ba3fe	fix(router): honor request fallbacks in the mid-stream continuation check (#50260)	2026-11-09 17:56:41 -08:00
paveln83	b28005fdf6	fix(router): merge default metadata under the request's instead of replacing it (#50700) ... * fix(router): merge default metadata under the request's instead of replacing it * fix(router): join default metadata tags after the request's tags A router default_litellm_params.metadata.tags list replaced the request's tags, so the deployment and credential tags the router adds never reached the logged request tags. Default tags now follow the request's tags, skipping ones already present, like the proxy's key tag merge * fix(router): keep masking a router default turns on Merging default metadata under the request let a caller's mask_input or mask_output false switch off masking an admin set in default_litellm_params.metadata. A masking flag the default turns on now stays on, and a caller can still turn masking on. Also gives this PR's test suppressions a reason and adds a missing blank line * fix(router): keep the router's cache namespace and drop malformed caller tags and dicts A caller's metadata.redis_namespace no longer beats a router default, so it cannot pick another tenant's cache partition. A caller tags value that is not a list, or a non-dict value under a dict default, is treated as absent the way the proxy's merge_tags treats malformed tags, so the router defaults still apply. * test(router): check that mask_output stays on when the router default turns it on * fix(router): keep a router default guardrail whatever guardrails the request lists * fix(router): keep the admin's settings for a router default guardrail when the caller lists it * test(router): check that a router default guardrail entry keeps the admin's settings * fix(router): keep a router default private Langfuse trace private whatever the caller sends * fix(router): keep a router default function such as langfuse_masking_function over the caller's value --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 17:23:52 -08:00
Nate Armstrong	eae60c0bc9	perf(router): skip unused deployment reads in the callback filter step and cache anthropic capability probes (#50641)	2026-11-08 17:59:55 -08:00
paveln83	141a580951	fix(proxy)!: price expanded wildcard /model/info rows as the model their calls route to (#49128) ... * fix(proxy): price expanded wildcard /model/info rows as the model their calls route to * fix(proxy): read wildcard routing through the router registry entry points * fix(proxy): price a wildcard row only when the router sends its name there Use the router's own route candidates for the listed name, so an alias, a routing group, a more specific pattern, or a team's own deployment keeps a wildcard row unpriced * fix(proxy): show no price on a wildcard row whose calls go to another deployment * fix(proxy): keep the listed price on catch-all wildcard rows and match the edit path to the read path * fix(proxy): price a wildcard row aliased to a sibling name as the alias target * fix(proxy): price wildcard rows through the caller's alias renames and show nothing for rows another deployment takes * fix(proxy): keep same-model wildcard row prices, skip deleted team aliases, use the caller's team * fix(proxy): keep live team aliases when skipping deleted team alias targets * fix(proxy): show no wildcard row price when the same-model deployment serving it sets custom rates * fix(model_info): recognize a replayed wildcard row on a worker that has not loaded the wildcard * test(proxy): feed wildcard discovery real provider data in model info tests * fix(proxy): price a wildcard row another deployment serves as the model its calls bill A row whose calls reach another deployment now shows that deployment's model and limits, and goes blank only when those deployments bill more than one model or declare their own rate. A name no route serves keeps its own price, and a replayed row stores none of the metadata it showed. Drop the router's per-call scan of every deployment for a team key, since the team and name indexes already cover those names. * fix(proxy): price a wildcard row aliased to an aliased group as the group its calls reach An alias target that is itself aliased onward listed the onward group's deployments too, so the row showed no price while calls billed the first target's model. The billed model now comes only from the route candidates * fix(proxy): price a wildcard row billed by another deployment without the wildcard's base_model A row aliased to another deployment priced as that deployment's model, but the lookup still swapped in the source wildcard's base_model, so the row showed a rate its calls never bill --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 19:08:22 +00:00
joshua-berri	43112547fb	feat(proxy): total_timeout bounds a whole non-streaming request (#50496) ... * feat(proxy): total_timeout bounds a whole non-streaming request A `total_timeout` body field or `x-litellm-total-timeout` header (the header wins) gives a non-streaming /v1/chat/completions, /v1/messages or /v1/responses request one budget for preparation, retries, moderation and post-call processing. Past it the proxy answers 408 and cancels the upstream call, and a retry whose backoff cannot fit ends early with the error it would have retried. A body value on a streaming or background request is refused with a 400, a header value leaves it unbounded. A timed out request is billed like a client disconnect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): ignore any total_timeout header on requests it cannot bound, and never interrupt pre-call admission A globally configured x-litellm-total-timeout that is malformed no longer fails the streaming and background requests it is ignored on. The deadline holds while pre-call admission runs, since admission reserves shared quota before it records it, and an expired deadline then ends the request before dispatch, where the failure path hands the reservation back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): a total_timeout cuts slow pre-call hooks short, holding only the rate limit reservation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 00:43:50 -08:00
joshua-berri	3c3f266322	fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count (#50486) ... * fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count A routed attempt cancelled mid-call (client disconnect with cancel_on_disconnect) kept its ITPM/OTPM reservation and its least-busy in-flight count, and retries sharing one logging object leaked the count of every attempt after the first. Each async attempt now runs under its own attempt id, so the attempt and its own callbacks release what it holds exactly once, in tasks a cancellation can't stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): settle an io-token reservation once even through a logged copy of its metadata /v1/messages and /v1/responses stash the reservation in litellm_metadata, which the logging object copies. A cancelled attempt refunded it, then the proxy's failure logging refunded the copy again. Each stash now carries an id, and a reservation already settled under that id is not settled again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep spent tokens counted, refund a previous attempt's stash, and fit every call in flight A request cancelled after its provider answered now keeps the tokens it spent counted instead of refunding them. A retry refunds the reservation its previous attempt left behind even though it runs under a new attempt. The least-busy reservation store holds twice the keys, since each call in flight now keeps one more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): never evict a settled reservation id before its counters' window has passed Settled ids lived in a capacity-bounded cache, so past 10,000 settlements in two minutes a late copy of an evicted reservation was refunded a second time. They now leave only when their window expires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): start releases eagerly and keep attempt settlement out of router_utils A release that never waits (an in-memory counter) now finishes before its caller goes on, as it did when the caller awaited it inline, so a websocket turn's usage is counted by the time its logging is flushed. Each reservation carries its own id, and the attempt settlement module moves to litellm_core_utils, since litellm/utils.py may not import from router_utils. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): count only finished answers as spent, and declare the least-busy redis family A stream's wrapper returning is not an answer: a stream cancelled before its first chunk now hands back its whole reservation. Only a non-streaming result marks its attempt as answered. The module level least-busy shared decrement carries its redis family, as every direct redis caller must. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:54:26 -08:00
Nate Armstrong	a3bf1f4399	fix(router): record retries and fallback hops past a request's first logged failure (#50445)	2026-11-06 19:48:25 -08:00
joshua-berri	de0b389205	feat(router): add the Switchyard stage router as an auto router strategy (#50407) ... * feat(router): add Switchyard Stage Router strategy * feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes ModelCall.respond and fail async. The classifier is now a tagged union of an LLM judge and a decision model, and decision calls go through the router's adecisions so any decisions provider works. Algorithm misbehavior and library errors route to the capable model instead of failing the request, and the session id plus client session headers are always passed so capable holds apply without a classifier Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep the Switchyard router inside the type-check gate The gate's environment did not install the optional switchyard extra, so every libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other optional runtime packages, and imports go through switchyard_rust.libsy, which ships py.typed. Step handling moves into helpers so no Final is rebound in the loop, and the dashboard API types are regenerated for the new routing fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): route Switchyard on every endpoint and reject unsafe router saves The router now passes each request's call type to routing strategies, and Switchyard converts the request to chat messages with the existing converter for that call type before translating it. /v1/messages turns with tool or thinking blocks used to fall back to the capable model, and now route on their signals. The Anthropic converter can keep tool_result.is_error on an analysis copy, so Switchyard sees tool failures there, while requests sent to models never carry it Saves of a strategy router that lists itself now fail with a 400 instead of a 500 with a dead row, the capable and efficient models must differ, judge and decision calls are not retried before failing open, the Switchyard routing fields are classified for redaction, and the edit dialog returns the saved router instead of the stale one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): catch self-routing on patch and rename, and trim Switchyard tests The self-routing check now reads the stored params through the decrypting helper and runs when a patch carries no litellm_params, so a PATCH that points a router at itself or renames it onto one of its targets returns 400 instead of a 500 with a dead row. The Switchyard tests share one transcript builder and router fixture, and reformatting noise in unrelated test files is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): make Switchyard a strategy in the existing auto router form Switchyard is now a card in the existing strategy picker. Picking it swaps the tier and classifier fields for the Switchyard fields, and the shared name, team and save button create it through one submit branch, the same flow as the other strategies. The separate add form, the extracted picker and the hidden-form swap are gone, and the edit dialog keeps its own Switchyard form The judge's session_affinity and message_hash_fallback settings are removed, since Switchyard rejects both for the stage router Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): accept partial tool semantics and show server errors on model edit The Switchyard tool semantics field rejected mappings that list only some categories, although the server accepts them. Model PATCH errors were reduced to a generic message, so it now goes through the shared API client like model create and surfaces the server text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): show the routing strategy picker up front in the auto router form Every strategy was hidden behind an Advanced configuration link, so the quick setup read as the only option and never said it creates a heuristic router. The picker now always shows with Heuristics as the labelled default, the quick setup sits under it, and picking another strategy opens that strategy prefilled with the chosen models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): show the server's reason when a model settings or API key update fails Both screens replaced the rejection with a fixed message, so a 400 such as a router routing to itself read only as a generic failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(router): route Switchyard on the Rust router core main replaced the Python router core with the Rust one, which left Switchyard routers registered nowhere. Register them with the other pre-routing strategies, hand each strategy the endpoint the request came through, and evict Switchyard routers on reload and delete like the rest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form Switchyard session state is keyed by a hash of the caller's key hash and session id, and client session headers no longer reach Switchyard unscoped. The add form carries the quick setup models into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the schema reads API-saved configs with nulls and structured decision instructions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(router): make the Switchyard test folder a package * fix(router): read the Switchyard session key hash only from the proxy's own metadata * fix(proxy): keep nested nulls as null when masking deployment params * fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies * fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:07:57 -08:00
m.torres	150f4b706b	fix(router): count embedding batches per input in pre-call checks (#49851) ... * fix(router): count embedding batches per input in pre-call checks Embedding input lists went through the Responses input transform, which calls .get() on each item, so every list-input /embeddings call logged "failed to count tokens" and skipped the context window check. Lists of texts, token IDs or token ID lists now count as their longest input, since an embedding model's context window applies to each input on its own * fix(router): keep empty and boolean input lists on the Responses count path An empty list carries no embedding evidence, and counting it as a zero token batch skipped the instructions and tools of an empty Responses input. Boolean lists are not token IDs * fix(router): honor disable_token_counter for token ID embedding input Text inputs already counted as zero through token_counter when token counting is disabled, while token ID inputs were still measured by length and could be rejected. The router reads the flag through its dynamic config like its other process settings * refactor(router): type the embedding count helper with a read-only sequence and final locals * fix(router): count embedding batches per input on the media-free text projection The per-input count now runs on the text projection from the embedding media split, so data URIs a deployment embeds as media keep the media path and only text items are checked against the context window. A caption next to an image is checked on its own, and a text-only deployment still counts an image data URI as text * fix(router): keep embedding images out of the pre-call count only on image backends, and skip tokenizing inputs that fit by bytes Media leaves the embedding count only when every deployment in the group declares image embedding input, so a text-only backend still counts a data URI as text and an image backend outside the TPM media list no longer gets a false 400. A text no longer than the smallest window in UTF-8 bytes counts as its byte length without the tokenizer, since a token is never shorter than a byte. * fix(router): keep embedding images out of the pre-call count when any backend in the group embeds images Mark Cohere embed-english-light-v3.0 as embedding images, as its v3 siblings are. * fix(router): check embedding text batches only with an exact OpenAI tokenizer litellm's default tokenizer counts non-English text several times higher than non-OpenAI embedding providers do, so counting batches with it refused inputs the provider accepts. Text batches on those groups now stay unchecked as at the merge base, without the error log. Main's per-provider media split and the cost map are restored, so strings and embedding media keep main's path. * fix(router): count long embedding batches in slices so chat counts do not wait behind them A text batch is now tokenized in slices of 65,536 characters, each its own offloaded call, so the FIFO count limiter can serve a chat count between slices. The no-OpenAI-tokenizer rule for text lists moved into the pre-call check, so RoutingContext.count_pre_call_check_tokens counts a text list by its longest text. * fix(router): read embedding media off the event loop and wait on the count limiter in the fairness test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 11:32:08 -08:00
simonp9	1f15088fc9	feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408) ... * feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes The native Codex catalog now lists the key's other chat and responses models after ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list refreshes on Codex's timer. A request for a model configured by exact name on another provider goes through /v1/responses without the ChatGPT bearer or account header, while ChatGPT names and wildcard-only matches keep the subscription passthrough. Reasoning items another provider produced are dropped before forwarding to ChatGPT. * fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while `codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native /backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no default, so the provider applies its own default exactly as `codex -m` does * fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may call /v1/responses, drop unpersisted reasoning ids on the passthrough when store is not true, keep any slug the caller's ChatGPT account lists on the subscription via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard precedence over a catch-all deployment. * fix(proxy): keep Codex's native route on the subscription when the account list is unknown A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the account's list is known and leaves it out. A failed catalog read is remembered for 60 s and the Codex version comes only from a User-Agent whose product matches the originator header. The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows, and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so Codex stops refetching or dropping its catalog cache. LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge decodes wrapped thinking blocks with their signature. The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the native route checks the chatgpt/ form of a name the key holds only through that group. A none reasoning effort is dropped only under drop_params. The guide covers app tools, every routing case, and the ETag, and the catalog logs how many models the byte limit left out. * fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API * fix(proxy): keep Codex native turns working when the key's model listing fails A listing callback that raises no longer turns chatgpt.com's catalog or responses answer into a 500. The catalog keeps upstream's body and ETag, and responses keep upstream's X-Models-Etag. Under drop_params, a `none` reasoning effort is now dropped only when a candidate deployment does not explicitly support it, so an alias to a model like openai/gpt-5.4 keeps it. The account slug list and its unavailable marker are keyed on the account id plus the caller's ChatGPT Authorization header, so a failed fetch with one bearer cannot turn off LiteLLM routing for another caller on the same account. The auth step checks the Codex route before importing the Codex module. * fix(proxy): keep Codex model switches working across LiteLLM providers On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400 Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows * fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the native route now hoists them without flattening, like OpenAI and Azure and the chat completions bridge. Other native Responses providers still get flat names, now built with the bridge's namespace_tool_name_map, and each function_call they return gets its tool name and namespace back in the non-streaming body and in streamed output_item.added, output_item.done, and completed/incomplete/failed events. Every other event passes through unchanged * fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools Codex sends its freeform exec tool as a custom tool on every turn, and xAI's Responses API answers 422 for custom, namespace and additional_tools shapes. On the Codex native route, a model whose deployments include a native Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now asks for the chat completions bridge with use_chat_completions_api, which already turns custom and namespace tools into functions, hoists additional_tools, and puts the namespace back on each returned call OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock Mantle still get it on their own Responses API with additional_tools hoisted. The route's namespace flattening and the response-side namespace restore are removed, since no path sends flattened names anymore * fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve A reasoning item another deployment produced, replayed with store false and no blob the target can decrypt, now reaches the target as its readable summary without id or status, and is dropped when nothing readable is left. OpenAI answered 400 on the status field and 404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment replays and stored conversations keep the item whole * refactor(codex): reach encrypted reasoning stripping through the router registry The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise every target must decrypt it * fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog * fix(proxy): drop Codex's none effort for an alias whose native entry names no default * fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop * test(proxy): map the pinned reasoning efforts field in the router registry stub * fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases Codex native routes now send a key to LiteLLM only when admin_only_routes allows it, a native alias with configured service tiers starts with no tier picked, and the account slug cache declares its Redis key family. * fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items * fix(proxy): check vector store grants on file_search tools inside additional_tools items A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input. * fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes * fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply * fix(guardrails): check additional_tools items next to top-level function tools in tool_policy ToolPolicyGuardrail read tools from the request body only when no top-level function tool existed, so a blocked tool defined in an additional_tools item reached Fireworks once the Responses transform moved the item into tools after the pre-call hooks. Check every list on each request, including namespace members inside items. * fix(router): serve decryptable_reasoning from the Rust core's optional checks The merge with main removed the Python router core, so the registry's host has no optional_callbacks and no routing context, and native Codex requests with encrypted reasoning raised RustCoreUnsupported. The filter now lives with the optional checks, which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core Also give the static litellm.Router a declared constructor. Since the Python core was deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...) is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change * fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures. * fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias * fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases Auth no longer asks ChatGPT for the account's model list when the bearer is missing or is the LiteLLM key, so a key sent as the bearer never reaches it. tool_policy now reads the members of a top-level namespace, which is where the native route's hoist puts them. A global alias now reads encrypted content affinity from its target group. A non-list top-level tools value and a non-string function name now reach the provider, which refuses them, instead of raising a 500. The catalog ETag no longer depends on the order a process iterates priced service tiers in, so workers and pods agree on it. * fix(proxy): list a team's models in the order the team set them get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker * fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models and /backend-api/codex/responses only when the JWT auth flow would admit a direct /v1/responses call for that model, so team_allowed_routes, admin_allowed_routes and RBAC route limits hold on the handoff too. The Fireworks Responses transform also leaves additional_tools input items in place when the tools param is not a list, instead of raising while hoisting * fix(fireworks): hoist additional_tools when tools is a tuple litellm.responses passes a tuple tools argument through to the Fireworks transform unchanged, so the non-list guard skipped hoisting and left the additional_tools item in the input. Treat tuples like lists and cover both shapes in the drop_params merge test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 11:14:47 -08:00
ryan-crabbe-berri	ec46a981fc	feat(model-hub): show each deployment's full prices in the model hub (#50145) ... * feat(model-hub): show each deployment's full prices in the model hub The model hub only showed one input and one output price per model, the highest across its deployments. Each model group now lists every deployment's prices, read from the same entry billing charges, including cached tokens, long-context tiers, flex, priority, batch and tiered ranges. Both the logged-in and public hub detail views render them in one table per deployment, labeled by provider only * refactor(ui): move model hub pricing breakdown text onto i18n Route every string the deployment pricing breakdown writes through t() under a new deploymentPricing group in en.json, plus the three hub headings that host it. The English on screen is unchanged * fix(model-hub): show Vertex regional prices and explicit cached reads as billed A Vertex deployment outside the global endpoint now shows its token prices with the regional uplift billing applies at the deployment's location. The explicit cached read price shows per 1M tokens on its own row instead of a raw per-token value. The breakdown also says that adjustments made per request, like off-peak discounts and regional or data residency surcharges, aren't included * fix(model-hub): show prices written as text the way billing reads them A price written in config.yaml as 3e-7 arrives as text. Billing turns it into a number, but the hub dropped it, so a deployment showed no price or the cost map's price instead of its own. The hub now reads numeric text for regular prices and size-based price ranges, and skips text that isn't a number * style(ui): format public model hub test * test(model-hub): name the deployment pricing fixture to stay within the inline object lint budget	2026-11-06 10:45:44 -08:00
ishaan-j-berri	1bcd11706b	fix(proxy): discover models for LiteLLM Gateway wildcard providers (#50374) ... * fix(proxy): discover upstream LiteLLM Gateway wildcard models * fix(proxy): use the typed shared client for gateway discovery * fix(ui): show the selected LiteLLM Gateway logo	2026-11-06 09:51:45 +00:00
yujonglee	b1b236826a	feat(redis): serve the proxy's common Redis call with Rust stores (#50170) ... * feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 06:01:11 +00:00
ryan-crabbe-berri	9a473fbbb0	feat(proxy): give Codex accurate model entries for every model in /v1/models (#49825) ... * feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release	2026-11-05 19:31:32 -08:00
Nate Armstrong	588c79618c	fix(router): give the rust core's priority scheduler the router's redis cache (#50265) ... The Rust core built its Scheduler without a Redis cache, so the priority queue lived only in each worker's memory and was never shared across workers or instances. Build it on the routing cache's Redis cache, as the Python core did	2026-11-06 08:50:42 +09:00
Nate Armstrong	e011a4ea9b	fix(router): do not cool down a deployment on a 404 from a response id lookup (#50262) ... The Rust core records every attempt failure against the routed deployment, so a client GET, DELETE, cancel or input_items call on a response id that the provider answers with 404 counted toward cooldown. The removed Python core never saw these failures. Exempt them in failure_exits next to the background poll 404 and video lookup exits.	2026-11-05 15:34:31 -08:00
Nate Armstrong	067f93340f	refactor(router): delete the python router core, the rust core is the only core (#50178)	2026-11-05 10:40:46 -08:00
Nate Armstrong	9752869d84	feat(router): add the Rust router core with the host as its only boundary (#49513) ... Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/), served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in the host-side litellm/router_rust/ package (steps/, invoke/, surface/) driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state in litellm/router_rust/surface/ (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. The crate records each attempt's outcome itself rather than through litellm callbacks. A streamed attempt is deferred: it counts its request when the stream opens, and the host's stream wrappers report the success with the final usage, or a failure after the first chunk, when the stream ends, as the Python core's callbacks do. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores, with nothing pending; only the names of the Python core's own outcome callbacks are asserted on Python alone. litellm.Router is now final: subclassing it raises TypeError at the class statement, and the machinery that kept subclass overrides working is gone. The Rust core never honoured subclass overrides, so they would have silently stopped running there. Custom routing goes through register_plugin(), settings through apply() and outcomes through observe(). This is a breaking change for anyone subclassing Router. Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries, max_fallbacks, allowed_fails, retry_after, timeout, the tag settings, routing_strategy_args and the fallback lists as given (the crate reads them as numbers, update_settings casts as Python does); a deprecated `model_list = [...]` write reaches the Rust core; a failed set_model_list is all or nothing on both cores (the deployments served before the call are put back and the error raised); config_deployments() is a generator and `del` of a plain attribute works on the Rust core. Least-busy pick reservations from main now work on the Rust core: its selectors share one reservation store, routed calls reserve and free their unstarted picks, and select() and batch cancel do not reserve. On the Python core a dropped upsert takes its name back off deployment_names, and get_deployment_order reads every numeric spelling as one int so mixed orders no longer raise TypeError. Rulings on the remaining differences, the Python core changed where it was the odd one out: sync picks and picks by deployment id apply the team filter as async picks do, so another team's deployment is refused; the fallback lists and default_litellm_params in RouterSettings are copies, so an in-place edit is ignored and settings change only through apply(); one health record with an unreadable timestamp drops out alone; an empty fallback entry is skipped; a bool cooldown_time is refused at construction, a bool or string cooldown_time on a deployment or a call falls back to the router default, and a failure status that no HTTP status can be is recorded as 500; an infinite or NaN cost and a whole number wider than 64 bits (ptu_shares) are refused on both cores with the same error (litellm/router_utils/deployment_numbers.py); and a /v1/messages stream whose provider iterator raises cools the deployment and retries like a non-stream call, with the Rust core waiting for that cooldown before its retry picks. Differences the proxy parity fuzz and its regression tests found on the Rust core: a file upload or a batch on an uploaded file falls back only within its own model group, as on the Python core, since the file lives with one provider (the host sends same_group_fallbacks_only and the walk skips targets in other groups); Decisions calls drop deployments whose provider has no Decisions config, and deployments that refuse the audio or videos a call carries, unless that would leave none, as the Python core does, while the retry count still reads the whole group; a file upload that fails on one order level is created on the next level's deployments rather than the first level again; and a response that cannot be converted (a text_completion body without a message) reports only its own traceback, so a rate limit handled by the attempt before it no longer turns the error into a 429, and both cores return 500.	2026-11-04 21:47:58 -08:00
Oliver Yu	733b776403	fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) ... * fix(least-busy): reserve the picked deployment so a burst of requests spreads out The in-flight count only went up in log_pre_api_call, which runs after selection and the pre-call hooks. Requests selected in that window all saw the same counts and landed on the same deployment (16 simultaneous requests over 4 deployments went as lopsided as 12/4/0/0 in the proxy). When the router selects for a request it is about to send, least-busy now records a reservation for the chosen deployment in the same synchronous step that reads the reservations, and log_pre_api_call turns it into the normal in-flight count. Selections without request kwargs (token counting, probes) do not reserve. A reservation whose request never reaches the API call expires after 30 s. The shared and per-worker counts and their success/failure decrements are unchanged. * fix(least-busy): prune expired and empty selection reservations The reservation map kept a key per model group forever, so a wildcard route with caller-chosen model names grew it without bound. Drop expired and empty entries on every write. Signed-off-by: Oliver Yu <tc3oliver@gmail.com> * fix(least-busy): keep pick reservations in the shared router cache Selectors that share a router cache dedupe to one registered callback, so only that instance saw log_pre_api_call. Reservations held on the other instances were never released and skewed picks for the full TTL. Storing them in the router cache's in-memory layer lets any selector release them, and the per-key TTL replaces the full-map rebuild on every write. Tests now hold the burst with a barrier instead of a sleep and use a manual clock object. * fix(least-busy): restart the reservation TTL on every write InMemoryCache keeps the TTL a live key was first given, so a reservation key that kept being rewritten still expired 30 s after its first pick and took the newer reservations with it. Drop the key before each write so the TTL counts from the latest one, and leave it deleted once it is empty. Signed-off-by: Oliver Yu <tc3oliver@gmail.com> * fix(router): count least-busy decisions calls and release reservations on cache hits Least-busy now reads the model group from litellm_metadata before metadata, so /v1/decisions calls that carry caller metadata are counted in flight and release their reservation at pre-call. A response cache hit releases its reservation instead of decrementing another request's in-flight count, and Responses WebSocket turn admission selects without reserving. The reservation store is reached through a typed protocol. * fix(decisions): count proxy /v1/decisions calls in least-busy's in-flight totals * fix(least-busy): keep reservations out of the shared router cache and count requests that start during a Redis read Least-busy now reads a call's deployment from the routing metadata the router sets, so /v1/decisions logging params stay as on main and failed decisions calls no longer start cooldowns. * style(router): wrap the least-busy reservation store line * fix(least-busy): lock the reservation read-and-update so threaded sync calls keep each other's picks Router.completion from a thread pool and sync success callbacks can read and rewrite a deployment's reservations at the same time, so one thread's pick or release could be lost. * fix(least-busy): release a pick whose call ends before reaching the provider A call that fails or returns before the provider handoff (a 400 for an unsupported param, token counting, a proxy-gate rejection without a start log, batch create) now drops its reservation instead of decrementing a count it never added. Router.select picks and batch cancel no longer reserve, since neither makes a call that logs. * test(least-busy): give callback fixtures a fixed call-start time The routing-group tests fed least-busy a finished call without the start time litellm stamps before counting it, so the count stayed up after the fix. Fixtures now use a fixed timestamp instead of the clock. * fix(least-busy): free tier-group picks, stop lookups reserving, and count only started attempts Reservations are keyed by deployment id, so a complexity, auto, adaptive or quality router frees its tier pick when the call starts. Each started attempt is recorded under its call id, and only a recorded start lowers the count when the attempt ends, so a fallback hop rejected before the provider frees its pick. Whether a pick reserves now comes from a context variable set by the routed call sites, so get_available_deployment and the deprecated async_get_available_deployment no longer hold a deployment, and custom routing strategies and Router subclasses keep their method signatures. * fix(least-busy): free a request's unstarted picks when it ends Each pick now carries its call id, so a start frees only its own call's pick, and the router frees every pick a request never started once that request ends. A fallback hop rejected before the provider no longer leaves its pick reserved for 30 seconds. * fix(router): free a least-busy pick when a vector store create ends * fix(router): free least-busy picks of calls made without a call id * fix(router): free the least-busy pick of a sync text completion * test(router): use fixed start times in least-busy fixtures * fix(router): free the least-busy pick of a sync generic call that fails before its provider * fix(router): read in-flight counts and start attempts under one lock so a busy deployment never looks free * fix(router): keep pass-through lookups from reserving least-busy picks * test(least-busy): wait for the helper thread to start before the timed join * fix(least-busy): keep realtime sessions out of in-flight counts --------- Signed-off-by: Oliver Yu <tc3oliver@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-05 00:30:12 +00:00
zoveki	784754ca48	fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863) ... * fix(azure): answer 401 when Entra rejects the deployment's credential A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500. * fix(azure): keep network failures to Entra a retryable 500 azure-identity wraps a transport failure in ClientAuthenticationError, so the 401 mapping now skips errors whose cause chain holds ServiceRequestError or ServiceResponseError * fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials * fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch maps the same rejection, and the A2A endpoint answers it with HTTP 401. * fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown * fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections ClientSecretCredential drops Entra's HTTP status, so throttling and transient codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable instead of mapping to 401. The A2A endpoint answers 401 only for an AuthenticationError caused by an Entra token rejection, and the router skips cooldown for a caller-sent Entra field only on Azure-family deployments. The azure/ Decisions route and the Bing grounding search tool now map an environment-credential rejection to the same 401. * fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents * fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting The fallback hop skips cooldown when Entra rejected a credential the caller sent, Prometheus counts a deployment's own Entra rejection as a deployment failure again, and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router. * fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message A caller-sent Entra field now skips the deployment's cooldown only when it differs from the deployment, its named credential, the router defaults, the AZURE_* environment variables and the default scope. An empty value counts as unset, so a deployment whose own secret is wrong still cools down when the caller's field changes nothing about the token request. The /azure passthrough's Entra 401 message goes through the same internal hostname redaction as other client-facing errors. * fix(router): compare a caller's Entra field with the value the deployment uses A caller-sent Entra field skipped the cooldown only when it matched no configured source, so a value equal to an environment default the deployment overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own and let a caller's rejected credential cool down a working deployment. The check now resolves the value the deployment would use without the caller, in order from the router defaults, the deployment, its named credential, the AZURE_* environment variables and the default scope, and skips the cooldown only when the caller's value differs from it. * fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged * fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider * fix(azure): keep the streaming Entra rejection inside the failed attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-04 20:44:55 +00:00
tavren27	b9ad61d4f6	fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) ... * fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call * fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests * fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500 * fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message * fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever. A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry. On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed * fix(agentic loop): bill retried attempts and clean up what failed or capped turns return The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints * fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try * fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run. The failure row bills every round the loop served before the failure, at any depth, and a later attempt that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the response cost headers adding up. The error response's x-litellm-response-cost reports the same amount its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before. A refused tool call now matches by id among the refused ids, or by name among refused calls that carry no id, so a name-only refusal next to an id refusal is no longer kept. * fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one * fix(code_interpreter): delete only the session sandbox that failed, never its replacement * test(code_interpreter): type the session sandbox test helpers * fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors * fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too. A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row. A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main. A code interpreter sandbox that fails while running the code now says the code may have run but its result is lost, instead of saying the code did not run. * test(responses): bill a retried MCP request each attempt on its own row again Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row * fix(spend): name a failed follow-up attempt the success record already bills With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off * fix(spend): count a billed failed follow-up's tokens on the success row * fix(spend): bill a failed follow-up on a fallback served from cache * fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown * fix(spend): keep a failed follow-up's billing state off the request-merged call details * fix(spend): record the cost breakdown of a successful agentic follow-up The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-04 02:00:33 +00:00
devin-ai-integration[bot]	d1e4b50d6a	fix(router): keep a deployment's token limits off the shared backend cost map key (#49952) ... * fix(router): keep a deployment's token limits off the shared backend cost map key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve catalog token limits on shared keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): use catalog for shared token limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drop the class attribute patch from the deployment max_tokens test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): avoid provider auth during catalog key resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 16:14:44 -08:00
norkelbyte	3b35ae340d	fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks (#49831) ... * fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks * fix(rate-limit): count embedding media as media only on models that embed it A data URI sent to a text-only embedding model is read as text, so it keeps the text estimate there. On models flagged supports_embedding_image_input, image and audio use the chat media estimates and PDF and video count at most the model's max_input_tokens. The same estimate now feeds project and key limits, the usage-based, latency and cost routers' deployment tpm, and the deployment itpm check. * fix(proxy): count embedding media only where the backend reads it as media Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as media per item. Cohere counts it as media only when the whole batch is media, and Voyage, OCI and unknown models keep the text estimate. File blocks carrying a data URI count like the bare URI. The router context window pre-call check and the batch limiter now use the same estimate, resolving each deployment or JSONL row model to its backends. The cost map gains supports_embedding_image_input on voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0. * fix(router): hold each embedding media item to the context window on its own A PDF or video data URI alone already reserves at most the model's max_input_tokens, but the context window pre-call check then ran the blanked input through the Responses transform and added a few framing tokens, so a PDF that Vertex accepts got a 400. When every input item is media, the check now returns the largest item's media estimate, since an embedding request embeds each item separately * fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped. * fix(bedrock): bill Cohere embeddings by Bedrock's input token count header * fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage The strict base64 check now scans the payload in place instead of decoding and re-encoding a full copy. Bedrock's billed token header is used only for image requests, so text-only Bedrock Cohere usage matches main. The deployment itpm estimator change is reverted, since the router never passes embedding input to that check. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 22:06:59 +00:00
norkelbyte	864b98acec	fix(bedrock)!: fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 (#49170) ... * fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused * fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success * fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row A realtime session's success event now reports status failure, with the same error_information the spend row stores, when the session recorded a provider failure in its metadata. Usage already streamed stays billed and the router still counts one cooldown failure The spend row builder now runs every row's error_information through sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction), so the realtime failure rows get the same treatment as the proxy failure hook, which no longer sanitizes on its own to avoid truncating twice Drop the selection-failure tag on the committed-session refusal, since the shared logging object already keeps a retried refusal out of cooldown, and pin the router's realtime success counting with an OpenAI realtime deployment * fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop * fix(realtime): count a router success only for a session that ended cleanly A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers * fix(bedrock): log a dropped realtime session's usage from its own logging state The failure event replaced the shared standard_logging_object while the queued success dispatch was still running its callbacks, so loggers received $0 and 0 tokens for a billed dropped session. * fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed * fix(rate-limit): settle each token reservation once when a request logs both a failure and a success A dropped realtime session logs a success for its usage and then a failure. After a success the limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing twice. A new reservation written in the same context after a released one now starts clean, so reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or 28 ITPM for 27 and 20 used * fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once * fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge * test(rate-limit): pin the split hold-release flags across nested MCP pre-calls * fix(realtime): count a relay that fails on its own as an abnormal end * test(realtime): pin the relay failure exit with a client frame the provider rejects --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:56:23 +00:00
Mateo Wang	b777dd6f14	fix(router)!: scope context fallback precedence to the requested group (#49779) ... * fix(router): scope context fallback precedence to the requested group * test(router): keep existing fallback test formatting * fix(router): preserve context errors after ordered backup failures	2026-11-02 22:12:42 -08:00
jesus-berri	4ede2c012f	feat(model_info): add supports_continuation capability flag for Gemini long decode continuation (#49500) ... * feat(model_info): add supports_continuation capability flag for Gemini long decode continuation * fix(router): keep a deployment's supports_continuation override on that deployment A supports_continuation value set in one deployment's model_info was written onto the shared provider/model cost map key, so every other deployment of that model reported it too. List it with the other per-deployment capability flags, and cite Google's long decode page next to the cost map assertion * fix(cost-map): flag vertex_ai/gemini-4-argon-cyber-permissive with supports_continuation Google's Vertex Gemini 4 Argon guide gives gemini-4-argon-cyber-permissive the same long decode limit as gemini-4-argon, and LiteLLM already continues its long requests, so the cost map now says so in both copies. * fix(router): count only a boolean true supports_continuation when building a model group --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:53:58 -08:00
Kaustubh Anil Jain	6f3251c7e3	fix(responses): serialize sequence_number on every bridged streaming event (#44944) ... * fix(responses): serialize sequence_number on every bridged streaming event * test(responses): cover sequence_number on bridged streaming events * fix(responses): read undeclared fields through dict accessors and number guardrail block continuations * fix(responses): scope dict reads that see undeclared fields to Responses stream events Moves the extras-aware __getitem__, get, __contains__ and items from the shared response base onto a new base for the 37 stream event classes, so usage, Gemini, Interactions and moderation objects keep reading __dict__ and Gemini responses keep _hidden_params out of .get * fix(router): renumber Responses stream events only after a fallback takes over A stream that never falls back now keeps its provider's own sequence numbers, repeats included, so a starting_after resume still matches them. After a fallback the wrapper continues above the highest number the client saw * fix(guardrails): keep Responses stream numbers unique when LLM Shield flushes held text The restorer's flush copied the last delta and kept its sequence_number, so a native stream repeated a number and a bridged stream sent the flush unnumbered. A flush now takes the done event's slot and later events move up by one, and a stream the provider never closed numbers its flush after the last event. Also pins the created_at in the streaming iterator test fixture instead of reading the clock. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:01:58 -08:00
Nate Armstrong	22d0b1e443	fix(router): restore four base behaviors the RoutingContext refactor changed (#49395)	2026-10-31 22:58:34 +00:00
moe-berri	580b9266a1	feat(chat): resume replies after browser disconnects (#49461) ... * feat(chat): resume background MCP runs after browser disconnects * feat(chat): background ordinary text conversations * feat(chat): configure background replies from Settings * fix(chat): address background response review findings * test(sdk): finish the HTTP smoke stream explicitly * fix(chat): preserve background response timing metrics * test(router): terminate the scripted completion stream	2026-10-31 14:43:24 -07:00