Narrows reportAny / reportExplicitAny hot spots in provider transformations,
proxy endpoints, integrations and secret managers by introducing TypedDicts,
Protocols and object-typed boundaries instead of Any, then ratchets the
budget ceilings down to match.
reportAny 14765 -> 14076, reportExplicitAny 4493 -> 4128, ANN401 387 -> 307
/v1/responses/input_tokens returned 200 with a count for an empty
"input" ("" or []), while OpenAI returns a 400 missing_required_parameter.
The route also went through optimistic budget reservation, which is only
released by LLM success/failure callbacks that a token count never
reaches, so every call leaked a reservation until TTL expiry and could
429 real traffic. Both routes plus the /openai alias now join
/utils/token_counter in the reservation exemption set.
Move the blocked-usage mapping for /v1/responses next to
blocked_response_usage in guardrail_translation utils, map bridged chat
prompt/completion tokens to Responses API input/output tokens, and let
raise_passthrough_exception attach the blocked response so post-call
guardrail blocks report real usage
* feat(router): make routing groups callable as virtual models and list them in /v1/models
* fix(router): traffic-scoped cooldown exemption, live model_names on delete, group-info cache invalidation
* fix(router): share one recognized-model predicate across proxy gates, resolve aliases in group cooldown, read metadata via the dual-bucket owner
* fix(router): close the gate and cache families for callable groups, strip member access_groups from group rows, prove cooldown wiring end to end
* refactor(router): cache materialized group rows under the model-group cache owner and drop the redundant wiring test
* fix(router): warn-and-shadow on group name collisions, name-level test coverage for group helpers, faithful router doubles in a2a and cursor tests
* test(router): pin group cooldown metadata across the retry path
reportAny 16720 -> 15482 and reportExplicitAny 5689 -> 5316 with real types only: no casts, no ignores, no new Any. Whole-tree basedpyright drops 2173 diagnostics with zero per-rule or per-file regressions. Budgets ratcheted: basedpyright -2173, ruff-strict -188, type-discipline -55
Cursor appends -thinking-<level> and -fast to custom model names when the
user picks a thinking level or fast mode, so a model configured as
claude-opus-5 arrives as claude-opus-5-thinking-xhigh-fast and fails
routing with no healthy deployments. When the raw name is not servable by
the router but the suffix-stripped base name is, rewrite the body to the
base model and carry the thinking level into reasoning_effort (chat
bodies) or reasoning.effort (Responses bodies), never clobbering an
effort the client already sent. Explicitly configured aliases keep
winning because the raw-name servability check runs first.
The type-discipline gate flagged 17 new mutable-collection annotations and 31
new mutable-collection constructions added by this branch. Replace raw dict
literals with the OpenAI SDK's TypedDict call forms, annotate read-only params
as Mapping/Sequence, precompute the custom tool call id set as a frozenset,
and accumulate streamed arguments as tuples. The few places where a plain
list/dict is a hard contract (pydantic response fields, fastapi route tags,
parsed request bodies, in-place tool call patching) carry reasoned mutable-ok
suppressions instead. Ratchet the ruff, type-discipline, and basedpyright
budgets down by the violations this branch now fixes on net
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.
B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.
ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
A chat tool-call dict was classified custom-vs-function with four different
spellings: the non-streaming parser required type == "custom", the streaming
Delta coercion also accepted a custom payload without type, and the stream
assembler required a type that later chunks never carry. The same payload could
be a custom tool call mid-stream, a TypeError on the completed message, and
silently dropped from the assembled message. is_custom_tool_call_dict() is now
the single discriminator (explicit custom type, or a custom payload present)
used by both parsers, and the assembler classifies from the accumulated custom
payload, matching how the deltas it consumes were classified.
The tool envelope converter picked one exclusive payload source: the nested
dict when present, else the top level. An empty nested envelope therefore
shadowed top-level fields and the normalized tool lost its name. Payload
extraction is now total over both locations, nested first, and an envelope
with no name anywhere passes through unchanged instead of being emitted
stripped.
Chat Completions nests a named tool_choice under its tool type while the
Responses API keeps it flat; ChatCompletionNamedToolChoiceParam and
ChatCompletionNamedToolChoiceCustomParam both mark the nested key required. The
messages arm normalized tool definitions but forwarded tool_choice at whatever
level Cursor sent it, so a flat {"type": "custom", "name": "ApplyPatch"} reached
OpenAI unchanged and was rejected while the tool defs beside it nested correctly
A tool definition and a named tool_choice carry the same envelope, so both now
convert through a single _convert_tool_envelope, and _normalize_tool_dialect
moves tools and tool_choice together on each arm. That covers all four cells of
{tool def, tool_choice} x {to chat, to responses} and removes the shape where
one field can be converted while the other is missed, replacing three helpers
with two and cutting 24 lines
Also restores the end-to-end assertion that a flat tool_choice reaches
chat_completion nested, which had been flipped to pin the passthrough behavior
- share one _CustomToolCallAccess mixin across the 4 new custom-tool
classes instead of hand-rolling dict access on each
- inline the single-use _nest_flat_chat_tools / _flatten_chat_tools_for_responses
list wrappers at their call sites
- drop _nest_flat_chat_tool_choice: it rewrote object-form chat tool_choice
into {type,custom:{name}}, a shape OpenAI rejects; real Cursor never sends
tool_choice on the messages arm, so pass it through unchanged
Three gaps from the bridge becoming a mainstream path for chat traffic.
The chat to responses message converter only mapped function tool_calls,
so history carrying the native custom tool calls this PR introduced
raised "tool call not supported" on follow-up turns; custom entries now
map to custom_tool_call items and their results to
custom_tool_call_output. The stream translator returned an empty delta
for output_item.done on tool items, which left the responses guardrail
handler's tool extraction permanently empty (dead on staging too, where
the built chunk was discarded); stateless callers now receive the
complete tool call while per-stream callers keep the suppressed delta
that prevents client-side duplication. Cursor routing keyed on the
presence of a messages key, so a null or empty stub next to a real
agent-mode input array picked the chat arm; routing now keys on
messages content
OpenAI enables reasoning by default for gpt-5.4+ (unset reasoning_effort
means medium server-side) and Chat Completions rejects function tools
whenever reasoning is on, so a tools request without an explicit
reasoning_effort 400d instead of auto-bridging to the Responses API; the
bridge heuristic now treats unset effort as reasoning-active and honors
the documented escape hatch by keeping explicit "none" on chat
completions. The cursor input arm also gains the mirror of the
messages-arm normalization: chat-nested tool envelopes, grammar formats,
and object tool_choice flatten to the Responses dialect before dispatch
Live Cursor Ask-mode captures show the shape dialects mix PER LEVEL: the
tool envelope arrives chat-nested while the grammar format inside it is
still Responses-flat, so a normalizer that pattern-matches whole-tool
templates misses every hybrid. The cursor arm now normalizes the envelope
level and the format level independently and idempotently, making it
total over the envelope x format matrix; a parametrized 8-cell test pins
every combination. The reference BYOK bridge was checked and forwards
chat bodies verbatim, so there is no prior art for these hybrids
Cursor's ApplyPatch is a grammar-constrained custom tool; the Responses
surface carries the grammar flat while chat completions wraps the same
fields in a grammar object, so the nested envelope from the previous
commit still 400d at OpenAI (tools[N].custom.format.grammar). Adds a
shared flat to nested format helper pair in prompt_templates/common_utils
used by the cursor messages arm and the chat-to-responses bridge, nests
flat Responses-style tool_choice objects on the cursor arm, flattens chat
custom tool_choice on the chat-to-responses bridge, and maps custom
tool_choice to function tool_choice on the responses-to-chat bridge to
match that bridge's custom-to-function tool downgrade
Cursor Ask mode sends chat bodies whose tools array mixes nested function
tools with flat Responses-style custom tools; the /cursor messages arm now
nests those before delegating, published via the request parsed-body cache.
Core chat parsing gains first-class custom tool call types mirroring the
openai SDK union: a single dict dispatch feeds the provider-dict sinks,
Delta dispatch stops both stream re-parse sites from silently swallowing
custom deltas, the chunk builder accumulates custom input for spend logs,
function-assuming consumers (json-mode gate, multi_tool_use repair,
helicone, lunary) skip custom entries, and the chat-to-responses bridge
flattens nested custom tools to the Responses flat shape
- delegate messages-shaped bodies to the standard chat completions handler
- strip chat-only stream_options before the Responses pipeline
- fix cursor_data_generator signature (request kwarg) and duck-type the
stream gate so router-wrapped streams convert instead of leaking raw
Responses events
- convert custom_tool_call items and events to chat tool_calls in the
streaming and non-streaming paths; remap streamed tool_call indices to
0-based sequential; accumulate raw and pydantic tool calls into one choice
- normalize generic pydantic output items through the raw-dict handler
The repo linted at 120 (E501, isort) but ran ruff format at 88 via a
--line-length 88 override in the Makefile and CI, leaving the formatter
and the linter disagreeing on wrap width. Drop the override so ruff.toml's
line-length = 120 is the single source of truth and reformat the tree to
match.
* Add support for websocket via codex
* Add model alias and creds support
* fix: skip cost tracking for WS session wrapper call types
The @client decorator on _aresponses_websocket fires async_success_handler
with result=None after the session ends. This triggered cost tracking errors
because standard_logging_object is never built for None results.
Per-turn costs are correctly tracked by individual litellm.aresponses calls
inside the session. The outer session-level logging obj should not attempt
cost tracking.
Fix: skip _aresponses_websocket and _arealtime call types in deployment_callback_on_success,
RouterBudgetLimiting.async_log_success_event, and _PROXY_track_cost_callback.
* fix: address Greptile review comments
Fix JSON injection: use json.dumps instead of f-string interpolation for model name in WS body.
Add 30s timeout for first WS frame to prevent unbounded connection resource tie-up.
Restore per-event model override in streaming_iterator; fall back to connection-level model when event omits it.
Strengthen regression test: inject alias into kwargs via _update_kwargs_with_deployment mock so the test would fail on un-fixed code.
* fix: handle nested response.create format in first-frame model extraction
When ?model= is omitted, the first WS frame can carry the model in either flat
format (first_event["model"]) or nested format (first_event["response"]["model"]).
The flat-only check would silently reject clients using the nested wire format.
Mirrors the same two-format logic in _build_base_call_kwargs.
* fix: don't force connection-level custom_llm_provider on per-event model overrides
If a client sends a different model per response.create turn, litellm needs to
re-resolve the provider from that model string. Forcing the connection-level
custom_llm_provider would silently route the request to the wrong backend.
Only inject custom_llm_provider when the per-event model matches the
connection-level model.
* refactor: extract WS model extraction into testable function
Pull the flat/nested model extraction into _extract_model_from_first_ws_event
so tests import and exercise the real function rather than a copy.
* fix: compare providers not full model strings in _inject_credentials
The model == self.model guard was too strict: same-provider model variants
(e.g., vertex_ai/gemini-2.0 -> vertex_ai/gemini-1.5 on one connection) would
lose custom_llm_provider, breaking routing when a custom api_base is in use.
Compare the provider extracted by get_llm_provider instead, so same-provider
variants still inherit the connection-level provider while cross-provider
overrides let litellm re-resolve.
* style: black formatting
* refactor: extract first-frame model resolution to fix PLR0915 (too many statements)
* Fix responses WebSocket first-frame validation
* fix: classify WS first-frame read errors and clarify cost-skip log
Distinguish client disconnects from server errors when reading the
responses WebSocket first frame, make the cost-tracking skip log message
accurate for session wrappers (which do carry a model), and resolve the
connection-level provider once per session instead of on every
response.create event.
* test: cover WS first-frame read errors and same-provider credential injection
Adds regression tests for the still-uncovered responses WebSocket paths:
the timeout, invalid-JSON and missing-model branches of
_read_ws_model_from_first_frame, plus the provider comparison in
ManagedResponsesWebSocketHandler._same_provider and _inject_credentials
(same-provider model variants keep the connection provider; cross-provider
models re-resolve).
* fix(responses-ws): fall back to explicit custom_llm_provider when connection model is unresolvable
When a WebSocket session is opened with a custom deployment alias that litellm
cannot resolve to a provider, _connection_provider was None, so _same_provider
returned False for every resolvable per-event model and the connection-level
custom_llm_provider was dropped. Use the explicitly-set custom_llm_provider as
the connection provider in that case so same-provider per-event models still
inherit it while genuinely cross-provider models continue to re-resolve.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
- Guard logging_obj for None when skip_pre_call_logic=True: raise ValueError
if litellm_logging_obj not in data, preventing AttributeError downstream
- Add model=None to common_processing_pre_call_logic call in endpoints.py
to match style of other call sites
- Add test verifying rate-limited request never receives polling ID
Move pre-call checks (rate limits, guardrails, budget) to run BEFORE
polling ID creation in the background streaming flow. This prevents the
edge case where a rate-limited request receives a polling ID that
immediately fails.
Changes:
- Add skip_pre_call_logic parameter to base_process_llm_request to allow
skipping pre-call checks (avoiding double-counting of RPM/parallel requests)
- Run common_processing_pre_call_logic before generating polling ID in the
responses API endpoint. If rate limits/guardrails fail, return error
immediately without creating a polling ID
- Background streaming task passes skip_pre_call_logic=True to avoid re-running
pre-call checks that were already done before polling ID creation
- Add tests verifying skip_pre_call_logic parameter works correctly
Fixes the edge case where polling_via_cache would return a polling ID
for a request that immediately fails due to rate limiting.
This follow-up to PR #16862 allows users to specify models that should use
the native provider's background mode instead of polling via cache.
Config example:
litellm_settings:
responses:
background_mode:
polling_via_cache: ["openai"]
native_background_mode: ["o4-mini-deep-research"]
ttl: 3600
When a model is in native_background_mode list, should_use_polling_for_request
returns False, allowing the request to fall through to native provider handling.
Committed-By-Agent: cursor
- Fix bug where model names without slash (e.g., 'gpt-5') couldn't
match providers in polling_via_cache list
- Look up model in llm_router.model_name_to_deployment_indices
- Check ALL deployments for matching provider (supports load balancing)
- Check custom_llm_provider first, then extract from model string
- Add comprehensive tests for provider resolution logic
Committed-By-Agent: cursor
- Create new background_streaming.py in response_polling/
- Update endpoints.py to import from new location
- Update __init__.py to export background_streaming_task
- Add tests for module imports and structure
Committed-By-Agent: cursor
