* fix(passthrough): parse Bedrock stream spend incrementally instead of buffering the whole response
Bedrock pass-through streaming kept every relayed chunk in memory until EOF and
then decoded, parsed and translated the whole stream again for spend logging.
Large or concurrent streams could exhaust proxy worker memory.
Sync and async passthrough wrappers now hand each chunk to a provider stream
collector as it is relayed. Bedrock decodes event-stream frames incrementally,
folds consecutive text deltas, and keeps only what stream_chunk_builder needs
for usage, tool calls and metadata. Text deltas are no longer retained in the
Bedrock and Anthropic stream decoders either. Providers without a collector
keep the previous raw-bytes behavior. Collector failures are isolated so spend
tracking can never interrupt the customer stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): assert the spend payload the collector builds instead of mock internals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): type the Bedrock collector helpers by the collector protocol instead of asserting the class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
W&B's serverless catalog grows faster than the registry names it, so a model
they ship today resolves as non-reasoning here until someone edits the cost map,
and the caller's reasoning_effort is dropped or rejected.
Add a wandb-reasoning-baseline capability rule to fallback_generalizations so any
wandb/ id the map has not described defaults to supports_reasoning. Rules lose to
exact entries, so mapped non-reasoning models such as
wandb/meta-llama/Llama-3.1-8B-Instruct are unaffected.
The rule carries no mode and no pricing, so cost stays on the standard unpriced
behavior and the deployment does not read as catalog-mapped to the router's
reasoning-effort resolver.
Claude-Session: https://claude.ai/code/session_01A6SkwJdfZUmkzfUkrEkqX8
A flat list[str] sent to voyage-context-4 is treated as independent inputs and forwarded flat with
enable_auto_chunking=True, chunk_size=32000, and input_type=document unless the caller already set
input_type=query. Caller-supplied params now override the defaults instead of being clobbered.
The voyage-4 family and voyage-context-4 cost map entries already exist on litellm_internal_staging,
and voyage-4-nano is not served by the Voyage API, so those additions and their pricing test are dropped.
get and post already take follow_redirects. put built the request and sent it
with the client default, so a caller uploading to a URL it did not choose had
no way to refuse a redirect. Same plumbing as the other two methods.
* fix(openai): drop tool schema regex patterns OpenAI's validator cannot compile
OpenAI validates function tool parameters with jsonschema's format checker,
which compiles every pattern with Python re. Claude Code's Artifact tool ships
an ECMA-262 pattern with \p{..} Unicode property escapes, so any OpenAI target
behind /v1/messages, /v1/responses or /v1/chat/completions 400s with
"Invalid schema for function 'Artifact': '...' is not a 'regex'" for every
model family. Drop only the patterns Python re rejects, keep the rest, at the
same seams that already flatten top-level combinators.
* fix(openai): walk only schema positions, iteratively, and drop regexes for every openai deployment
Review round: the regex sanitizer now walks JSON Schema applicator positions
only (properties, items, prefixItems, combinators, $defs, additionalProperties
and the rest), so a pattern key inside default, examples, const or a vendor
extension is data and stays. It also drops patternProperties keys Python re
cannot compile, which OpenAI checks the same way. The walk is level-order and
rebuilt deepest level first instead of recursive, so the code-quality recursion
gate passes and there is no depth cap below what a JSON parser admits. On the
chat wire an openai deployment with a custom api_base now drops such regexes
too, since that base is usually a proxy in front of the same validator, while
the lossier combinator flattening stays limited to api.openai.com hosts.
Encode complete non-Claude source names and include source_model in the
Claude Code listing. Preserve configured route and alias precedence,
normalize once before model policy checks, and select CLI models using
explicit source identity instead of name stripping or positional joins.
Resolves LIT-7360
Claude-Session: https://claude.ai/code/session_01WyqeRhfZGm26zAnHx9P3kq
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The async-transform path in the shared handler sent every provider's sign_and_log to the AWS pool, so Ollama, Snowflake, and watsonx queued behind Bedrock refreshes there. Only SignsRequestsWithAWS configs take run_aws_signing now, the rest keep the default-executor hop they had. The executor isolation test also runs on its own loop instead of pinning a one-thread default executor on the session-scoped pytest loop
asyncio.to_thread puts every Bedrock signing on the loop's default executor, the same pool every provider's async entry point hops through, so signings parked on botocore's refresh lock queued unrelated providers behind Bedrock. run_aws_signing runs them on a 16-thread pool only AWS signing uses
The encrypted_content_affinity check only read the pin from the Responses
input, which /v1/messages builds after the router has picked a deployment,
so a model group spread across OpenAI orgs sent follow-up turns to the
wrong org and got invalid_encrypted_content back. The check now also
decodes the pin from bridge-tagged thinking and redacted_thinking blocks
in the Anthropic messages. The bridge also keeps a deployment's own
include list next to reasoning.encrypted_content instead of replacing it.
Guardrails that hand back tool_calls in their own shape (vendor JSON, user code output) raised a KeyError on the non-stream Responses write-back. Returned tool calls are now validated before comparison; a shape or count that does not line up leaves every tool-call item unchanged and logs a warning naming the guardrail. A tool call's name is written back only when the guardrail changed it, so a nameless custom_tool_call no longer picks up the custom_tool placeholder.
The converse reasoning gate only matched openai.gpt-5, so gpt-6-astra fell through to
Anthropic's thinking block and Bedrock rejected the call with 400 Unknown parameter:
'thinking'. Match any openai.gpt-<digit> model at the three gate sites instead.
Nova 2 lite and pro accept forced tool_choice on Converse (verified live on
us.amazon.nova-2-lite-v1:0), so the nine Nova 2 registry keys now advertise
supports_tool_choice. The invoke dispatcher also forwards json_mode to Nova like it
already does for Anthropic and TwelveLabs.
cancel_batch resolved credentials and polled job status without the
deployment's aws_session_tags, so on a tag-gated trust policy batch
create and retrieve succeeded while cancel failed with AccessDenied.
Thread the tags through both calls and cover it with a regression test
that fakes STS behind a tag-gated trust policy.
initialize_azure_sdk_client now falls back to litellm.constants.DEFAULT_MAX_RETRIES
when litellm_params carries no max_retries, so off-router Azure clients (files,
batches, fine-tuning, assistants, audio) honor the env var like OpenAI clients do.
Router paths already default max_retries to 0 and are unchanged.
Regression tests cover the default, explicit 0/5/None values, and the env var
reaching the SDK client in a fresh interpreter.
