* fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions
Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema
* fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens
Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema.
* fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models
Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them.
* fix(bedrock): drop tool custom object on the Mantle count_tokens fallback
Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way.
* fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages
* test(vertex): annotate count_tokens tool test locals with Final
* fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names
* fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module
Session scope booted one 2-worker proxy per xdist worker and kept it until the
worker's session ended, so a CircleCI node holding two of these files piled up
proxies until the VM stopped responding. Module scope lets the harness group
each file's rig tests onto one worker and tears the proxies down after them.
* docs(tests): keep proxy fixtures module scoped in integration tests
* docs(tests): say how the harness groups proxy fixtures in the integration AGENTS.md rule
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): resume background response streams with stream and starting_after
Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}
Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally
* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams
A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.
* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve
A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.
* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params
* fix(responses): keep deployment stream off retrieves and reject oversized cursors
* fix(responses): keep router default stream off plain retrieves
* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing
* test(responses): replay a real stream in the billing test and type the fake wrapper chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn
* fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards
* fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set
* fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit
* fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum
* fix(responses): leave a turn without a phase when its content message cannot be read
* test(responses): add black box tests for chat turn history replay
* test(responses): expect the replay record on bridged chat replies
* test(responses): expect the replay record in bridged chat translation cases
* fix(streaming): keep the served service tier on the finish chunk the stream adds
* test(responses): read the expected chat message without JSON validation so the replay matcher stays
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(health): report openai realtime deployments with a missing or invalid api key as unhealthy
* fix(health): time out a silent openai realtime handshake and keep the first-event parsing in the openai handler
* fix(health): treat invalid_model as healthy, read the first event only on OpenAI hosts, map rate limits to 429
* fix(health): respect the global api_base, report a missing key as 401, keep invalid_model healthy only for known models
Drop the separate first-event timeout so model_info.health_check_timeout governs the wait, and keep the shared stream error table unchanged by raising AuthenticationError for invalid_api_key in the realtime check
* fix(health): keep invalid_model healthy unless the cost map lists the model outside realtime, and load realtime types lazily
* fix(health): read the cost map's listed endpoints, not its mode, since registering a realtime deployment overwrites the mode
* fix(health): read a realtime gateway's first event and keep Live deployments listing /v1/live healthy
Behind a custom api_base the OpenAI realtime health check now waits up to
3 seconds for the first event, so a gateway that passes on OpenAI's
invalid_api_key error reports unhealthy while a silent gateway stays
healthy. /v1/live counts as a realtime endpoint for the invalid_model rule.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add BigModel provider with native Chat, Responses, and Messages APIs
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
* test(providers): make BigModel unit tests a package
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
* docs(providers): clarify BigModel parameter interface
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
* fix(bigmodel): keep the Coding Plan base, surface Responses auth errors, and scope routes
Chat Completions keeps a GLM Coding Plan api_base. Responses raises the status from
BigModel's success=false body instead of returning an empty 200, and a JSON body
answering a stream request is replayed as a one-shot stream. Messages streams keep
ping and [DONE] lines. bigmodel leaves openai_compatible_providers so speech, image,
and transcription routes send it no request
* fix(responses): replay a JSON body on a stream request only when it has a Content-Length
A gateway that labels a chunked SSE stream as application/json kept streaming before
the replay, so only a sized body is read in full
* fix(bigmodel): scope the JSON stream replay to BigModel, map its business error codes, and echo requested tools
* fix(bigmodel): forward Codex's include and parallel_tool_calls, keep the OpenAI org header off BigModel, and pass unchanged Messages stream events through
* fix(bigmodel): map disabled thinking to low effort, keep Messages temperature, raise context window and content policy errors, and default parallel_tool_calls to true
* fix(bigmodel): keep the caller's reasoning_effort when Messages rewrites disabled thinking
* fix(bigmodel): keep context-window and content-policy errors on streamed Responses replays
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover Codex tool lists repeated across turns on the Bedrock chat bridge and Mantle
* fix(responses): keep each Codex tool once when its tool list repeats across turns
* test(responses): type the new tool list dedupe unit tests
* fix(proxy): list every priced Vertex family under a vertex_ai/* grant
A vertex_ai/* key listed none of the 68 cost map rows priced under the
Mistral, Llama, Qwen, AI21, OpenAI, image, video and embedding labels
because their sets were never part of the vertex_ai union, and Qwen rows
had no ingestion branch at all. Wildcard expansion also dropped a meta/
or openai/ publisher segment as if it were a provider prefix, and the
wildcard health check would have picked $0-per-token rows first once the
union grew, so it now sorts paid rows ahead of unpriced ones. The
VERTEX_FAMILIES_A_VERTEX_WILDCARD_GRANT_DOES_NOT_LIST allowlist is gone
so the cost map guard enforces every label from now on.
* fix(health): route every wildcard health-check pick through the listed provider
* fix(health-check): keep per-character-billed rows ahead of rows priced $0 everywhere in the wildcard pick
* fix(health-check): break $0-per-token ties toward per-character-billed rows in the wildcard pick
* fix(health): only health-check ids the router sends to the provider
The wildcard health check picker now skips ids that get_llm_provider routes to a different provider or to none, so the longer vertex_ai union cannot pull partner rows into the check. Drops the forced custom_llm_provider and the per-character tie-break from earlier revisions, and pins the replicate and wandb wildcard listing to each row's full owner id.
* fix(health): match wildcard health picks against the provider's own route
The previous check kept a bare id only when get_llm_provider routed it to the
exact provider name, which dropped every ai21 row because bare ai21 ids route
to the ai21_chat alias, so ai21/* health checks sent no request.
Compare the bare id's route with the route of the provider-prefixed id
instead, so aliases match. Resolve routes with inferred_provider, which
returns the declared provider for github_copilot and chatgpt without
starting their device-code login.
* fix(health): keep wildcard health picks that only route through a saved custom_llm_provider
Rank picks routed to the checked provider first and ids routed to no provider
second, each by cost, and skip only ids routed to a different provider, so a
wildcard saved with custom_llm_provider still sends its health request
* refactor(vertex_ai): mark the new Vertex registry sets Final
* chore: re-run CI against current main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost_map): mark fireworks_ai/minimax-m3 as vision and pin capabilities verified against live calls
Fireworks' serverless minimax-m3 answers image inputs, but its model
listings say supports_image_input false, so the last two registry syncs
flipped supports_vision back to false and the Fireworks pre-flight turned
every image request into a 400 without ever calling the provider.
Flip both minimax-m3 rows back to supports_vision true and record the
live-call evidence in ci_cd/cost_map_pins.json. The cost map guard now
fails any PR that lowers a pinned capability, quoting the evidence, and
the sync bot may not touch the pins file.
* fix(cost_map): fail the guard when the pins file is missing
* fix(cost_map_guard): skip the pin checks on a branch cut before the pins file existed
* fix(ci): check cost map pins against the merge result and keep price sync PRs off the pins file
* fix(ci): run the cost map guard against the live base branch tip and re-run it when main's pins change
* fix(ci): pass the live base tip to the cost map guard through env and re-run every guard run on a pin change
main now runs the guard on pull_request with the base branch's copy of the script, so the workflow passes the
live tip in COST_MAP_GUARD_TARGET, which the older script ignores. The recheck reads past GraphQL's 100-file
cap, re-runs every guard run for a shared head, and fails when a re-run cannot start. The guard tells a merge
conflict apart from a failed merge-tree call.
* fix(ci): run the cost map pins recheck on litellm_ base branches too
* fix(ci): bar bot-opened PRs from the pins, recheck pin PRs on cost map changes, and skip the merge for untouched PRs
* fix(ci): read the pushed files from the compare API in the cost map pins recheck
The push event GitHub hands to Actions leaves each commit's added, modified and
removed lists null, so the recheck never saw a pins change and re-ran only the PRs
that touch the pins. Compare the push's before and after commits instead, and
re-check every cost map PR when the branch is new, the push was forced, the
compare fails, or it lists 300 files, the most it returns.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(token_counter): move the o200k provenance note into the assert message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust_bridge): azure/gpt-4o now resolves to o200k_base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know
* fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer
* fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer
A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model.
* fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model
* fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k
A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count
base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response
Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model
* fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written
Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted.
* test: type the new token-counting tests' parameters
* fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it
* fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test
* perf(token-counter): classify each distinct served model once when walking a model group
Budgeted requests walk every deployment in their model group, since the
router's cached copy is outside the router surface the proxy may use.
Classifying each distinct served model once instead of once per
deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms
with the same result.
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): stream every citation batch before completion
* test(responses): cover citations arriving before answer text
* fix(responses): close reasoning before a citation opens the message item
* test(responses): build citation batches without rebinding locals
* fix(responses): hold citations until reasoning closes
Citations that arrive while reasoning streams now wait and stream once the message item opens, so a citation no longer ends reasoning early. Annotation events also stop sending sequence_number, matching the text deltas.
* fix(responses): keep held citations when reasoning ends without answer text
A citation that arrives before the answer message opens now waits for the
message, then streams once text arrives or at stream end when the final
response has a message. A disconnect keeps the held citation in the billed
response, and a tool call with no answer text no longer streams a message
item that response.completed lacks.
* fix(responses): keep a held citation's message ahead of the next tool call
When the answer already has content, a citation held before the message opens now opens it before the next tool call takes its output index, so the stream and response.completed keep the message ahead of the function call like the non-streaming response.
* test(responses): simplify citation comprehensions
* test(responses): set the new citation state on the hand-built streaming iterator
* fix(responses): keep held citations ahead of tool calls and in the final message
A citation that arrived before a tool call with no answer text was dropped from response.completed and the non-streaming response, because the final transform built a message only when content was set. Keep an empty-text message for those citations, and flush a held citation before the next tool call opens, so the streamed message stays ahead of the function call.
* test(responses): annotate the cited tool call test locals as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* build(deps): suppress filelock Windows advisories and drop stale certifi entry to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): suppress dev-only msal-node advisory in vscode-extension until azure identity allows 7.1
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump ddtrace and msgpack to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump pylint to 4.0.9 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): lock dnspython 2.9.0 on Python 3.11+ and defer 3.10 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump qs in vscode-extension to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): extend the pillow 12.3.0 override for Python 3.10 to 2026-11-27
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): extend the diskcache ignore, drop the withdrawn braces entry, and name the filelock cap in its reasons
* build(deps): bump langgraph and add dated ignores for the 2026-10-28 filelock advisories
* build(deps): move tornado to 6.6 on Python 3.11+, name huggingface_hub's filelock use, and scope a grype rule to the filelock High
* build(deps): say which packages import tornado in its ignore reasons
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover team and key vector store allowlists when no vector store registry is loaded
* fix(proxy): enforce team and key Allowed Vector Stores when no vector store registry is loaded
* fix(proxy): skip non-string vector_store_ids entries instead of failing auth
* fix(proxy): answer 400 on a malformed vector_store_ids in legacy mode too
Legacy vector store access checks now parse the request the same way vector_store_deny_by_default does, so a vector_store_ids that is not a list of strings is a 400 with or without a registry.
* fix(proxy): honor drop_params and a tool's string vector_store_ids in the legacy vector store check
Under drop_params the check reads the stores registry retrieval uses instead of answering 400, and retrieval reads only the string entries of a top-level list. Without it, a tool's single string id is checked as one store, the way the registry reads it.
* fix(proxy): honor deployment and router drop settings for a malformed vector_store_ids
* test(proxy): drop vector store registry mocks the access check no longer reads
* fix(auth): answer 400 for a malformed vector_store_ids only under a key or team allowlist
* fix(auth): match managed vector store ids by their provider store and read a string id as one store
A LiteLLM-managed store id now matches an allowlist entry for the provider store it decodes to, in both
directions, so an allowlist naming store A no longer refuses A's managed id. A top-level vector_store_ids
string, which is how ?vector_store_ids= arrives, is checked as one store id instead of answering 400.
Registry retrieval reads an SDK tuple of ids again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(credentials): pin that PATCH keeps the stored secret when it echoes the masked read-back
* fix(credentials): keep the stored secret when PATCH echoes its masked read-back
* fix(credentials): only treat a mask on a key GET masks as a placeholder
PATCH now skips mask-shaped values only on the keys get_masked_values masks, using the masker's own keyword check, so a value like api_base ab****cd is saved as sent again. Also drops the new test docstrings and tightens the new test helper types
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio
The drain regression guard compared the wall-clock time of draining 80,000 values against 40,000 and tripped its 3x bound whenever the CI runner was busy. A cursor drain allocates only per-value decode scratch while every re-slice of the remainder copies it, so the guard now bounds peak allocation during the drain, which does not move with runner load and still fails on every quadratic variant
* test(json_fragment_accumulator): widen the drain's peak bound to a quarter of the payload
* test(json_fragment_accumulator): guard copying drains by peak memory and rescanning drains by CPU-time scaling
* test(json_fragment_accumulator): move the drain's timing guard from the unit suite into a CodSpeed benchmark
* test(json_fragment_accumulator): measure only the pops after the buffer is joined, so a drain that copies once passes
* test(json_fragment_accumulator): separate every other streamed value with a space, so a copy after whitespace fails the drain test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover management and health routes labelling rejections as auth failures
* fix(proxy): name management and health route errors by their status instead of auth_error
* fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls
/key/list and /key/aliases now share the catch-all helper too, so they stop
adding the Authentication Error prefix and typing a 400 as internal_server_error
* fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error
A real outage surfaces from prisma's query engine as a builtin AttributeError, which the
prisma-module check skipped, so the route catch-alls still answered 500 or 400
* fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status
* fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503
* fix(proxy): answer 503 for database failovers and label provider auth errors like chat
A database that refuses connections while starting up, shutting down, in
recovery, or read-only after a failover now answers 503 no_db_connection
instead of 400. A provider's 401 or 403 keeps the type /chat/completions
gives it, while a 401 or 403 the proxy raises keeps auth_error. An
exception with no message is named by its type.
* fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause
* test(proxy): expect the OpenAI error shape from management route refusals
Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): normalize Google system parts and null tools
* fix(count_tokens): preserve typed validator return
* fix(count_tokens): validate single system part without typing.cast
* fix(count_tokens): accept every Google spelling of a repeated field
Google countTokens now reads a repeated field sent as one object, null, or a
list with null items the same way as its list form, for system parts, tools,
and function declarations, matching what Vertex accepts.
* fix(count_tokens): read every Google spelling of contents and skip null system text
contents and each content's parts now read one object, null, and null items like their list form, the same as system parts and tools. A one-object parts used to translate to no messages, so non-Google models counted it as empty. A system part whose text is null is skipped, as Vertex AI does.
* fix(google_genai): read null-text media system parts as media and one-object content parts on generateContent
* fix(google_proxy): count null-text system parts that carry only part metadata, like Vertex AI
A system part with null or missing text and only metadata such as thought or thoughtSignature, or an empty part, now counts as empty instead of answering 400. Parts that carry an image or file with null text still answer 400.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(context_management): count the system prompt toward the clear_tool_uses input_tokens trigger
Anthropic's API compares an input_tokens trigger with the whole prompt, system
included; the polyfill counted messages and tools only, so it cleared late by the
system prompt's size. compact_20260112 already counts it: share its system_to_text.
* fix(context_management): take read-only types in the system prompt count
The type-discipline gate (LIT001) rejects new list and dict annotations.
* chore: rerun CI (proxy-build-from-pip timed out)
* fix(context_management): count the system prompt through compact's shared counter
Finishes the main merge's conflict in clear_tool_uses.py. The input_tokens trigger now adds the system prompt through count_system_tokens, the same counter compact uses, while the count reused for cleared_input_tokens stays messages and tools only, so the reported savings are just the cleared tool results
* perf(context_management): skip the token count when nothing can be cleared
The keep check runs before the trigger count, the system prompt reader accepts any mapping block, and the system prompt test gets short parameter ids
* test(context_management): keep the off-loop count test counting after the keep check moved first
* test(context_management): cover a client compaction summary counting toward the trigger
The trigger also skips the system prompt count when messages and tools alone already pass it
* fix(context_management): move a client compaction summary into the system prompt when clearing runs before compact
* fix(context_management): clear only the history after a client compaction block so compact still replaces its summary
* fix(context_management): keep tool calls beside a client compaction block clearable
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): enforce an end user's per-model budget from their linked budget
The shared auth checks now refuse a request once the end user's spend on
that model reaches the model_max_budget of their budget (a linked budget or
the default applied to them), for virtual keys and the master key alike.
The cap is read from the end-user row, since the master key's cached auth
object keeps whichever end user's limits it was cached with. The per-model
end-user check inside virtual-key validation is removed: it read a token
field that is never set there, so it never ran.
* fix(auth): enforce the end user's per-model budget on JWT auth and skip it on budget-exempt routes
JWT tokens now carry the end user's per-model caps so spend is charged against them and over-cap requests are refused. Routes that skip budget checks, like /spend/calculate, skip this check too. The proxy default end-user budget is covered by a test.
* fix(auth): write the end user's per-model cap onto every token and check it after the member fallback
The shared auth checks now resolve the end user's per-model cap, falling back to the key's or the proxy's default end-user budget when the end user has no row yet, and write it onto the token so the spend hook charges the cap the check enforces on every auth path. The check runs after the team-member budget fallback, on the model it picked, and the JWT builder's duplicate check is removed
* fix(auth): keep only a per-model end-user cap that custom auth itself set
A key-auth token reached through custom auth's auto hand-back now takes the current end user's cap, so a cached master-key or virtual-key object no longer carries an earlier end user's cap into another end user's request.
* test(auth): build the custom-auth token with model_copy instead of setting the flag in place
* test(auth): hold an end user to their own or their key's budget cap when the proxy has no default budget
* fix(proxy): give each request its own copy of a custom-auth token so end-user caps and ids stay per request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): give lite codex aliases of a Codex-known model that model's catalog entry
lite codex built its Codex model catalog from /v1/models ids only, so a proxy
alias of a model Codex knows (api-gpt-6-luna for openai/gpt-6-luna) got the
fallback entry. That entry has no supports_search_tool or tool_mode, so Codex
sent every MCP and app tool inline and each turn cost far more input tokens.
lite now reads each listed model's upstream from the proxy's /v1/model/info
(team_public_model_name when set, else model_name, first deployment wins) and
passes it to the shared catalog builder, the same way the proxy's own Codex
route does. If that request fails, the catalog is built as before.
* fix(cli): give a lite codex alias an upstream only when all its deployments route to one OpenAI model
* fix(cli): cap a lite codex alias's context window at its listed max_input_tokens and count only OpenAI-hosted deployments
* fix(cli): keep Codex's tier, upgrade and reasoning defaults off lite codex aliases
* fix(cli): keep Codex's own effort, verbosity and promo off lite codex aliases and skip credential-hosted deployments
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed
* fix(responses): drop a null created_by from replayed input items too
A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null.
* fix(responses): drop a null created_by from counted input items too
* test(responses): cover saved reasoning nulls and re-compaction of a saved conversation
* fix(responses): drop a set created_by from input under drop_params
* fix(types): type the Responses output serializer's parameters
* fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* docs(adr): add architecture decision records, starting with provider usage extras transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(adrs): correct where provider usage extras survive and scope the live proof bar to runtime changes
* docs(adrs): say which chat transforms keep unknown usage keys and where a new field is passed
* docs(adrs): correct ADR 001's account of the xAI billing PR, date its code facts, and move the billing recipe to CONTRIBUTING.md
* docs(adrs): say the stream builder carries Anthropic's inference_geo by name and drops other extras
* docs(adrs): cover /v1/messages, the cost_per_token arm, and the Gemini tier side channel in the usage-field ADR and contribution guide
* docs(adrs): decide that usage extras reach callers and shared paths copy them, scope out provider-reported cost, and fix the proof steps
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(azure_ai): cover Foundry agents on the project Responses API
* feat(azure_ai): reach Foundry agents through the project Responses API
* fix(azure_ai): keep system prompts and project-endpoint auth on Foundry agent calls and follow-ups
* fix(azure_ai): read AZURE_SCOPE before the Foundry audience default on project endpoints
* fix(azure_ai): reject or drop instructions on previous_response_id turns and a caller's other agent for Foundry agents
* fix(azure_ai): apply the other-agent rule to agent_reference sent at the top level of a Foundry agent request
* fix(azure_ai): match the caller's agent by name and version, and hide the agent's own instructions
* fix(azure_ai): keep instructions on plain Foundry models that return a null agent_reference
* fix(azure_ai): keep a Foundry agent version requested at the top level when extra_body also names the agent
* fix(azure_ai): answer 400 for /responses/compact on a Foundry agent instead of calling the project with the agent name
* feat(ocr): support outputContentFormat=markdown for Azure Document Intelligence
* fix(ocr): map only span endpoints when slicing Azure Markdown pages
* fix(ocr): accept any casing of outputContentFormat and report code point offsets
Azure reads outputContentFormat case-insensitively, so Markdown and TEXT
now map to the same values instead of returning 400. When the proxy asks
Azure for code point offsets to split Markdown pages, the response now
carries stringIndexType so callers know the unit of table and key-value
spans. Tests use rstest and the enum derives VariantArray per the Rust
workspace rules.
* fix(ocr): ask Azure for code point offsets only on prebuilt-layout and keep line-joined pages when spans are malformed
* fix(ocr): trim Azure DI outputContentFormat, accept output_content_format, and drop invalid values under drop_params
* fix(ocr): fall back to output_content_format when outputContentFormat is empty
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit
* test(unit): share the short socket fixtures across unit tests
Move socket_dir and socket_path to tests/unit/conftest.py so the datadog
redis test's absent socket fails for the right reason on a Mac, rename
the pgbouncer helper's directory argument, and record the rule in
tests/unit/AGENTS.md
* test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path
socket_dir now uses /tmp when TMPDIR plus the longest socket suffix
would pass macOS's sun_path limit, and its teardown ignores errors so a
failed test reports only its own failure
* test(unit): keep socket fixtures inside tmp_path behind a short link
socket_dir now links a short temp directory to tmp_path, so artifacts keep
pytest's retention and a killed run leaks only an empty link. The fallback
to /tmp counts the temp directory in bytes, and socket_tempdir points
tempfile at it for the pooler tests whose socket directory production picks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): reset_time picks when deployment rpd and tpd daily limits start over
reset_time is "HH:MM" in UTC next to rpd/tpd in litellm_params or model_info. The daily cache keys are dated by now minus that offset and the daily 429 retry-after counts to the next reset, so a free tier whose provider quota resets at 08:00 UTC comes back at 08:00 instead of midnight. Without it the window stays the UTC calendar day.
* chore(ui): regenerate schema.d.ts with reset_time
* refactor(router): rpd, tpd and reset_time are read from litellm_params only
* feat(router): rename reset_time to daily_reset_time, drop the request-sent value, accept only HH:MM
* chore(ui): regenerate schema.d.ts for daily_reset_time
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): list TypeSafe Jev models in the admin model picker
* fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids
Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones.
* fix(typesafe): match only published Jev ids and keep picker loads off the secret manager
Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file.
* test(typesafe): type the picker test fixture and tests
* fix(typesafe): match only published Jev versions and cover non-version suffixes in tests
* fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker
* fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers
The dashboard sends include_model_access_groups on every /models call, so
gate the TypeSafe names on wildcard routes too and skip access-group-only
listings. Also say the typesafe-jev-ids rule matches case-insensitively.
* fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main
A typesafe/* router deployment registers typesafe/* in the cost map, so the
allowlist pickers offered a bare *, which allows every model. The pickers now
list only priced names the rule sends to TypeSafe. The non-expand list also
stops de-duplicating every entry and only skips TypeSafe names already listed.
* feat: add StreamLake provider metadata
* fix(streamlake): complete provider discovery and test request flows
* test(streamlake): assert cost and tool-param invariants instead of pinning vendor prices
* fix(streamlake): accept tools and reasoning on endpoint ids and match the current price page
Endpoint ids have no cost-map row, so tools and reasoning_effort were rejected
before reaching StreamLake. The provider now declares both capabilities, and
named non-reasoning rows opt out explicitly.
The rows now follow StreamLake's 2026-09-17 price page: drop the KAT-Coder and
Qwen3 rows it no longer serves, add the served models that had no price, drop
account-level rpm and tpm, and fill missing context windows. StreamLake is also
registered for streamlake/* wildcards and its messages and responses support.
* fix(streamlake): regenerate the endpoint matrix and type the new tests
The backup endpoint matrix advertised chat only while the root map listed
Messages and Responses, so /public/endpoints hid both. Regenerating with
scripts/generate_provider_endpoints_support.py makes the two files match.
Types the StreamLake tests and sets the Add Model test's API key with
fireEvent.change, per the dashboard test guide.
* fix(streamlake): drop the retired max_tokens field from the StreamLake cost-map rows
* fix(streamlake): price the four served models StreamLake publishes rates for and add the Qwen3.8 cache prices
* fix(streamlake): send token caps as max_completion_tokens and mark image models
StreamLake ignores max_tokens on kimi-k2.6, kimi-k2.7-code, qwen3.6-27b and
qwen3.6-35b-a3b, while every served model honors max_completion_tokens, so
the JSON provider now maps max_tokens to max_completion_tokens instead of the
reverse. Rows whose models read an image in a live probe set supports_vision,
and the Qwen3.8 rows drop the explicit cache write price, which this route
never uses since cache_control is dropped
* fix(streamlake): keep cache_control marks, price explicit cache reads, and use API input limits
StreamLake reads cache_control, so the provider keeps the marks through a preserve_cache_control flag. Qwen3.8 explicit cache hits now use their own read price, rows with a cache read price declare prompt caching, three models StreamLake serves without reasoning drop reasoning_effort, and max_input_tokens follows the limits the API reports.
* fix(utils): warn when a custom_provider_map name matches a built-in provider
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): skip embeddings input in google_text_moderation and openai_moderations
Since v1.103.0 the proxy dispatches these legacy moderation callbacks on every route, and they read the embeddings input field as text, so benign embedding inputs were blocked. Return early for embedding call types, matching the Aim and Cato guardrails.
* test(guardrails): type the embeddings moderation test parameters
* fix(guardrails): keep openai_moderations moderating embeddings, skip only google_text_moderation
Google's moderation scores topic categories like Finance and Health, so it blocks
documents being indexed by subject. OpenAI's moderation scores harm categories only,
so it keeps moderating embedding input as it has since v1.103.0
* fix(guardrails): skip Gigachat passthrough embeddings in google_text_moderation
Treat an allm_passthrough_route call whose endpoint ends in embeddings as
non-conversational, log the skip, return None from the early exit, and type
the hook's data so the basedpyright gate does not grow
* fix(guardrails): match only plain embeddings passthrough paths in google_text_moderation
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): scan Responses tool results in texts-mode guardrails
* fix(guardrails): keep guardrail-added tools and non-text parts on Responses tool scans
Tool result texts now come from the same conversion that builds the chat tool message, so items without a call_id, dict outputs, and outputs with no text parts stay aligned for Bedrock and PANW. A changed joined text replaces the first text part and keeps the other parts. scan_only_tool_results appends tools the guardrail returns with new names, like chat, and tool_result and computer_call_output items are scanned through the shared type list.
* fix(guardrails): skip empty and screenshot tool outputs and keep text part types on Responses scans
* fix(guardrails): keep Bedrock latest-turn masking aligned when tool outputs send no text
* fix(guardrails): scan each Responses tool output text part on its own
* fix(guardrails): split a Responses tool output's structured row when only textless outputs share its call_id
* fix(guardrails): pair each Responses tool output with its own structured row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop metadata from compact requests to OpenAI and Azure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop user and drop_params-rejected fields on compact and route Azure compaction to v1
* fix(responses): keep stream on compact and keep non-dated Azure api-versions on the v1 compact route
* fix(responses): drop compact params only for providers verified to reject them
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): copy the stream's hidden response_cost into the client's streamed usage.cost
The stream is priced once into hidden response_cost (stream builder for chat and text completions, the Responses iterator for the terminal event) and the proxy egress copies that bill into the streamed usage.cost on a copy of the event. Inside the pipeline usage.cost stays what the upstream reported. Spend rows do not change; the billing fix for upstream-reported usage.cost follows in the next PR cut from #48287.
* fix(proxy): type the bridge bill carry and the builder's reported-cost guard for the lint gates
* style: format the chat-over-Responses bridge helper signature
* refactor(bridge): build the billed Responses event with model_copy instead of a TypeAdapter chain
* fix(stream): keep writing the computed cost into usage.cost as on main, the hidden copy is only for the proxy egress
* fix(stream): write litellm's bill into the usage frame's usage.cost instead of a hidden field the proxy copies
* test(stream): keep main's formatting in test_streaming_handler.py, only the new test changes
* refactor(stream): drop the docstring on _stamp_usage_cost, the body says it
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): stop dumping the request body at every layer under --detailed_debug
The proxy dumped the full request body at every layer it passed through
(proxy entry, pre-call, router, SDK call echo, param mapping, logging
object), so one Codex "hi" wrote over 500KB of debug log. Keep the inbound
request line and the outgoing provider request, summarize large values in
the SDK call echo, and stop per-request lines that report nothing
(callback disable checks with no disable header, cost map name probes,
response id decode misses, vector store path misses).
* fix(logging): summarize tools in the param mapping debug lines
Non-Default params and Final returned optional params still printed full
tool definitions, so a Codex-style request repeated them twice more. Use
the same size summary as the SDK call echo and cover tools in the test.
* perf(logging): build the param mapping debug summary only when debug is on
* fix(bedrock): omit system prompt from audio transcription
* fix(bedrock): report Converse token usage on audio transcription
Bedrock transcriptions returned usage null and were billed at $0. Map
Converse inputTokens, outputTokens and totalTokens into the transcription
usage so spend uses the model's per-token price. input_token_details is
optional because Converse does not split audio and text tokens.
Also switch the updated request-shape test to rstest.
* fix(bedrock): default transcription temperature to 0 and price ap-southeast-2 Voxtral regionally
Voxtral wrapped some transcripts in quotes at its default sampling
temperature. In direct Converse calls in ap-southeast-2, 0 of 48 calls at
temperature 0 were framed against 7 of 72 at the default, and instruction
wording made no clear difference, so the request now sends temperature 0
when the caller sets none, matching OpenAI's transcription default.
Calls in ap-southeast-2 billed at the us-east-1 price. The cost map gains
bedrock/ap-southeast-2 Voxtral rows with all four service tiers, the Bedrock
transcription response carries the deployment region in its hidden params,
and the transcription token cost uses the region-resolved model key.
The bridge now builds the response with model_copy instead of assigning
usage after construction, and the native wheel test checks the usage the
compiled route returns.
* fix(bedrock): forward transcription language, reject unsupported formats, and bill env and Tokyo/London regions
get_optional_params_transcription had no Bedrock branch, so language,
response_format and timestamp_granularities were dropped before the Rust
transform ever saw them. Bedrock now forwards language, prompt and
temperature, accepts response_format json or text (the only shapes it can
return), and answers 400 for srt, vtt, verbose_json or
timestamp_granularities unless drop_params is set.
The billing region now falls back to AWS_REGION_NAME and then AWS_REGION
when aws_region_name is unset, the same order the Rust resolver uses to
pick the endpoint, so an env-configured region gets its regional price.
Adds ap-northeast-1 and eu-west-2 Voxtral rows (standard, batch, flex and
priority) from the AWS Bedrock offer files, the other two regions AWS
serves Voxtral audio in.
* fix(bedrock): send transcription temperature as a number and name the rejected param
The proxy keeps multipart form values as strings, so -F temperature=0.2
reached Converse as "0.2" and Bedrock answered 400. The Bedrock mapper now
converts temperature to a float and answers 400 (or drops it under
drop_params) when it is not a finite number.
get_optional_params_transcription copied prompt and temperature back from
the raw call after the Bedrock branch ran, which undid that conversion and
any drop. The Bedrock branch now owns every OpenAI transcription param.
An unsupported param such as timestamp_granularities now gets a 400 that
names it instead of the generic "user/encoding format" message.
* refactor(bedrock): return unsupported transcription params as a value and raise at the param step
* fix(bedrock): drop the retired max_tokens field from the regional voxtral price rows
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): stop log lines from leaving terminal colors on
* test(proxy): drop source scan for unclosed terminal colors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): ask OpenAI and Azure audio chat models for audio output in the health check probe
* fix(health): check the probed model when health_check_model is set, and mark Azure audio models as audio output
A deployment with an audio base_model and a text health_check_model got the audio fields sent to its text probe model. The Azure gpt-audio and gpt-4o audio preview cost-map entries now set supports_audio_output, so a dated Azure name like azure/gpt-audio-1.5-2026-02-23 gets the audio probe too
* fix(health): read litellm_params.base_model, and drop the stored base_model when Test Connect swaps the model
* fix(health): judge the audio probe on a swapped-in health_check_params model, and keep Test Connect's stored base_model
* fix(health): look up a provider-prefixed base_model under its own provider and keep text-only operator modalities
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages
Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes.
* fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body
Vertex reads the API version from the body alone, so the caller's
anthropic-version header now becomes the body's anthropic_version instead
of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party
versions keep vertex-2023-10-16, and the header itself is still never sent.
Same change on the Python proxy and the Rust gateway.
* fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept
Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are
carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no
header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under
drop_params and answers a 400 naming drop_params without it, on the Python proxy
and the Rust gateway.
* refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place
* fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python
The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there
* docs(rust router): name Router::from_config in the crate README
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): audit #47445
Integration cells for billing every round of a streamed MCP auto-execute on one spend row, on a native Responses upstream and the chat bridge, across tool rounds, a rejected or broken follow-up, a post-call guardrail block, a client hang-up, the response cache, and the next bridged turn
* test(responses): check a broken follow-up bills its own nonzero usage
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46997
Add request-path cells for the global proxy spend counter: a key with no user spends toward litellm max_budget, and two proxies sharing Redis add every request to one counter that matches the proxy budget row
* test(proxy): wait for the recorded budget spend and shared counter before checking them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
OpenAI shut down dall-e-2 and dall-e-3 on 2026-05-12. Drop their image
generation and edit configs, the Azure dall-e-2 create-and-poll flow, and
the silent dall-e-2 fallback when no model is passed.
image_generation without a model or provider now fails locally with
"model is required". image_edit and image_variation leave model out of
the provider request instead of sending dall-e-2, so providers that
ignore model (Topaz) keep working and OpenAI returns its own 400.
* ci(zizmor): run zizmor 1.31.1 without ignores and own the workflow
zizmor-action has no input for --no-ignores, so the workflow now installs the
zizmor 1.31.1 release binary directly, pinned by SHA-256 like grype in
image-scan.yml, and runs it with --no-ignores and --no-config. Inline
`zizmor: ignore` comments and config-file ignores no longer hide findings.
Behavior otherwise matches the old action defaults (regular persona, online
audits with the job token, medium and above, GitHub annotations)
Adds a CODEOWNERS entry for .github/workflows/zizmor.yml
* ci: run spend-logs-guard and cost-map-guard on pull_request
Both guards ran on pull_request_target behind a zizmor ignore comment. They
now run on pull_request and check out the base commit by sha, so the guard
scripts still come from the base branch while the PR's head is only fetched
and diffed. The zizmor ignore comments are gone
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.
Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream
The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.
SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF
Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed
Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.
Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.
Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.
* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests
Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.
* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines
* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter
* test: type the iterator class parameters in the line-separator stream tests
* fix(responses): stream tool call arguments opened in a text chunk
* fix(responses): hold a tool call that shares a chunk with text until the text ends
* test: type the a2a SSE handler helper and the sandbox fake responses
---------
Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
