GPT reasoning-series models (gpt-5.x, gpt-6) on Bedrock Mantle chat completions now send max_completion_tokens, carrying a deployment or client max_tokens over with its value. Every other Mantle chat model keeps sending max_tokens.
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e
* test: fix disk cache collection, ungate saturation check, drop RSS memory tests
* test: assert one delivered log per completion in Datadog and LangSmith checks
* test: drop SDK-direct e2e copies and align cost-map path filter cases
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: migrate selected llm translation contracts
* test: migrate anthropic completion translation tests
* test: migrate bedrock and cohere translation tests
* test: tighten bedrock translation migration
* test: migrate azure and openai model translation tests
* test: migrate openai and small-provider translation tests
* test: migrate gemini and vertex translation tests
* test: migrate optional params and rerank translation tests
* test: migrate huggingface, json and openrouter translation tests
* test: fix openrouter integration expectations
* test: migrate databricks, xai, cometapi and fireworks translation tests
* test: migrate ovhcloud, prompt and small provider translation tests
* test: split reasoning effort grid into unit and e2e
* test: migrate router and realtime guardrail translation tests
* test: migrate google interactions translation tests
* test: migrate a2a, evals, skills and hosted vllm translation tests
* test: remove remaining llm translation suite
* test: tighten migrated llm translation unit tests and land missing rows
Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover
* test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP
Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths
* test: close lost-intent gaps found in review of the llm translation migration
Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised
* test: assert provider acceptance in the reasoning effort e2e
The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid
* test: cover async and stream-end paths the legacy translation suite exercised
Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker
* test: type the new llm translation e2e calls and mark moved fixture dirs as packages
* test: name the reasoning effort e2e models as module constants
* test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name
* test: block the prompt the shared guardrail denies and drop the reasoning presence check
* test: drop the duplicate litellm import left by the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move guardrails, image, audio, batches, search legacy tests to unit
* test: restore the custom guardrail and remaining chunk 1 unit rows
* test: move the ocr matrix to e2e
* test: drop comments from moved tests and fix vertex ocr credentials
* test: move litellm utils and unified google legacy tests to their tiers
* test: move logging callback and pass-through unit legacy tests to their tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move store_model_in_db legacy tests and start the admin ui migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: finish the proxy job rows and drop their CircleCI jobs
Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI
* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names
* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist
* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in
* test: type the e2e beta header, prompt cache and secret manager values
* test: assert the missing model error message on the OpenAI routes
* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks
* test: make the langfuse fixture directory a package for the layout check
* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane
* test: keep the ocr e2e pdf fixture next to the test
* test: put meta below covers on the streaming tool search e2e
* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures
* test: count only OCR calls in the OCR matrix recorder
* test: pin OCR parse bodies and billed cost, group new e2e cases into classes
* ci: restore the python image anchor dropped in the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): call the ai_decide AI Function route for decisions and the auto-router
Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not
on a serving endpoint's /invocations route. The decisions provider and the
auto-router classifier now accept only the ai_decide model, post to that route
without a model field, unwrap the response envelope, and map a noul answer's
probability to the System One noul field.
* test(databricks): bound the megabyte model name test by scaling, not wall time
* test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound
* test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route
* fix(databricks): keep serving endpoint decisions beside the ai_decide route
* test(databricks): assert the whole classifier request body in the test routing case
* test(databricks): correct System One spend call type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan or refuse attachments in the bedrock guardrail
* fix(guardrails): cover native pre_call and computer screenshots in the bedrock attachment scan
* fix(guardrails): classify every attachment shape the bedrock guardrail can receive
* test(guardrails): type the attachment-scanning test guardrail
* fix(guardrails): keep base behavior for Bedrock subclasses and malformed messages
- skip the attachment scan when a subclass overrides apply_guardrail
- ignore non-string message role/type instead of raising
- treat a null Converse document/video/audio block as absent
- keep the base request-body debug log for text-only calls
* fix(guardrails): let text documents, unpadded base64 images and request-override subclasses through as before
* fix(guardrails): let Converse text documents through and refuse documents that carry images
* fix(guardrails): scan images inside content documents instead of refusing them
* fix(guardrails): scan text document content with the Bedrock guardrail
* fix(guardrails): scan document title, context and nested documents
* Expand nested content documents without recursion
* Keep attachment scan within the type-discipline budget
* Take the image format and pixel size from the image bytes
* chore(guardrails): satisfy type-discipline gate after merging main
* chore(guardrails): drop docstrings added by the attachment scan
* fix(guardrails): scan Converse text documents and guardContent images
A Converse document whose source is text (or a list of text parts) now
goes through the Bedrock text scan instead of being refused as
unscannable, so the pass-through request that worked before this PR
keeps working. A Converse guardContent image is now scanned like any
other image. Byte and S3 document sources stay unscannable.
Adds the integration test that drives every scanned endpoint through a
real proxy against a scripted Bedrock ApplyGuardrail stub, and retypes
the touched hook signatures for the type gates.
* test(guardrails): cover Bedrock checks mode in the attachment integration test
Adds a checks-mode proxy to the rig: images are refused without calling Bedrock on every route, and a Converse text document is sent to InvokeGuardrailChecks.
* fix(guardrails): scan a Converse document's name and context whatever its source
A Converse PDF or S3 document let through by skip_unscannable_attachments now still has its name and context scanned as text, the same as the Anthropic document path.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test: migrate local_testing SDK and provider tests out of build_and_test
* test: cover sagemaker region, hf template and nova paths end to end
Region rows now go through litellm.completion on the sagemaker/ handler that
the legacy tests exercised, the prompt template test mocks the Hugging Face
tokenizer config instead of switching to custom_prompt_dict, streaming tests
assert role, finish reason and body, and Nova single turn and streaming run
through litellm.completion with a recorded request body
* test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot
Split the catch-all providers migration file by behavior. The fake endpoint
route tests now hit the owned upstream directly, as the legacy harness tests
did. LM Studio asserts the outbound key, path and stripped model. The embedding
timeout test asserts the proxy maps a deployment timeout to a 408 and forwards
the read timeout to the provider client, instead of timing out the test client
The Azure streaming tool-call test also asserts the second identical stream is
served from the Redis cache with the same reassembled tool call
The Docker network test always skipped in CI because the security job has no
litellm/litellm image and no egress to pull one, and it tested a published image
rather than the checkout. It now boots the checkout proxy with every outbound
HTTP connection refused and asserts readiness, the bundled cost map fallback and
that the cost map host is the only egress attempted
* test: tighten the migrated e2e rows for local_testing SDK and provider tests
Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by
the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main.
Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers
so the header actually reaches the provider, and mark the creation-token case 1h then
5m like the legacy test did
Reuse the Bedrock file's own registration for the pydantic response test, swap the
hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer
names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in
markers the same way as the other opt-in markers
* test: restore lost assertions in migrated test_main tests
Restore the role alternation check in test_completion_anthropic_hanging, the
embedding and streaming response-header checks, the logit_bias drop check and
the gemini system and safety settings checks. Add an azure 429 retry-after
test for test_exception_with_headers, trigger the router fallback with
mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s
* test: restore lost legacy assertions in migrated vertex ai unit tests
* test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard
* test: strengthen migrated anthropic, azure, bedrock and cohere provider tests
Assert full request bodies and usage instead of presence checks, restore the
dummy-tool and custom_llm_provider assertions the legacy tests made, add the
async and per-model variants the legacy parametrization covered, split
streamed tool arguments across deltas, and move the Azure chat streaming
tests to the azure.py mirror file
* test: tighten migrated llms provider tests and cover legacy intents lost in the move
Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt
* test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients
* test: restore single request assertions in the deep_infra and azure_ai embedding tests
* test: drop a no-op streaming cost flag from the openai streaming cost test
* test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke
* test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard
* test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e
* test: send the vertex maps e2e request through the openai sdk
* test: drop the local_testing jobs that select no tests after the migrations
* test: wait for the streamed cache entry and keep custom Mistral rates test-local
* test: keep untouched Mistral test lines as they were
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(sagemaker): keep AWS credentials out of the sagemaker_chat request body
sagemaker_chat and sagemaker_nova built the container body from every optional
param, so aws_access_key_id, aws_secret_access_key, aws_session_token and the
other aws_* auth params, plus model_id and sagemaker_base_url, were sent in the
JSON body to the endpoint container. The request is still SigV4-signed with the
same credentials, which stay in optional_params for signing
* fix(sagemaker): limit the sagemaker_chat body filter to aws_ params
model_id and sagemaker_base_url are not credentials, so they stay in the
body as before and the change sticks to what the fix is about
* test(sagemaker): assert the whole sagemaker_chat request body
* fix(sagemaker): filter the sagemaker_chat body with the shared AWS auth param list
* test(sagemaker): cover the sagemaker_chat body on every endpoint, credential source, and outage
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): parse Vertex timestamps as UTC so created_at is not shifted by the host timezone
convert_vertex_datetime_to_openai_datetime parsed the trailing-Z Vertex and GCS
timestamps into a naive datetime, so .timestamp() read them as host local time
and every created_at it produced was off by the host UTC offset (about 7h ahead
on a PDT host). Attach UTC before converting, and accept the whole-second and
nanosecond forms Google's RFC 3339 timestamps also use, which used to raise.
Refs LIT-9383
* test(vertex_ai): type the UTC epoch test, guard tzset, build the rig config without mutation
* feat(sdk): add litellm.systemone and trim litellm.decisions to the OpenAI shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run tests/unit/systemone in the core-utils unit shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): keep litellm.decisions(state=...) working with a DeprecationWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make tests/unit/systemone a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): surface the litellm.decisions(state=...) deprecation as a UserWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): guardrail Bedrock Converse stream text per content block
The buffered converse-stream guardrail joined every text delta of the same
kind across all content blocks into one string, guardrailed it, then split the
result back over the frames by length. When a guardrail edit changed the
length (a mask), characters moved between blocks: text spilled from one text
block into the next, and parallel toolUse inputs became invalid JSON.
Group fragments per (contentBlockIndex, kind) so each block is guardrailed and
written back on its own.
* test(guardrails): run the Bedrock stream block tests through the real post-call hook
Drive de_anonymize_event_stream with a real ProxyLogging and pass-through logging object, with a fake guardrail at the boundary, and assert the texts the guardrail receives as well as the frames the client gets back
* test(guardrails): pin the logging start time in the Bedrock stream tests
* Add ScaleDown model pricing
Registers the five ScaleDown models in the pricing map. ScaleDown bills on
input tokens only, so output_cost_per_token is zero for every entry. The
classify and decisions models bill at $0.04 per million input tokens, and
extract, summarize, and compress bill at $0.05 per million.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.04 per million and drop unverified context limits
Every ScaleDown model now bills $0.04 per million input tokens. The
32000-token max_input_tokens/max_output_tokens values were not confirmed
against the service, so they are removed rather than registered.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.05 per million input tokens
This is the standard rate in ScaleDown's billing and matches the usage dashboard.
The Decisions usage.cost field is not used for pricing.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add regression tests for the ScaleDown pricing entries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cite the source of the ScaleDown rate in the pricing tests and type them
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add ScaleDown as a chat provider
ScaleDown serves five models over two wire protocols. extract, summarize, and
compress speak OpenAI chat on /v1/chat/completions, while classify and
decisions implement the Jev Decisions API on /v1/scaledown, taking a state
object plus a questions map and returning one typed answer per question.
The provider authenticates with x-api-key rather than a bearer token, and the
decisions models need a non-chat request body, so it routes through the HTTP
handler instead of the OpenAI passthrough. Malformed decisions requests are
rejected locally with a message naming the problem, so callers do not have to
decode an opaque 422 from upstream. Decisions bills one model call per
question, so the cost upstream reports is passed through rather than
re-estimated from token counts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Route ScaleDown domain models to the native API and stop trusting usage.cost
The /v1/chat/completions and /v1/models routes return 403 on api.scaledown.xyz,
so extract, summarize and compress now call the native /extract,
/summarization/abstractive and /compress/raw/ endpoints. The default api_base is
the bare host; a trailing /v1 is stripped and re-added only for decisions.
The Decisions usage.cost field was about 600x the dashboard amount, so it is no
longer passed through; cost is computed from input tokens at the registered rate.
Native responses are returned unchanged as JSON, including the _value/_span_anchor
wrappers on nested extraction. They carry no output token count, so
completion_tokens is reported as 0 and documented as unmeasured.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Move ScaleDown translation into llms/ and address review findings
- Dispatch moves to llms/scaledown/chat/handler.py; main.py keeps a thin call
and get_llm_provider_logic.py no longer has a ScaleDown branch, since the
config resolves key and base URL itself.
- questions passed via extra_body are merged and validated after the shared
merge (sign_request), so the documented extra_body path works.
- extra_body can no longer change the upstream model, and decisions state may
carry a document but not text, so guardrails always see the text.
- A custom api_base requires its own api_key; the environment key is only sent
to the default host or SCALEDOWN_API_BASE.
- Image content parts become the decisions document; max_completion_tokens maps
to max_tokens; stream is served as one chunk for every model.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Refuse prompt-bearing extra_body keys and empty-key fallback to the env credential
extra_body is merged after guardrails inspect the messages, so each model now
accepts only option keys there (extract threshold/top_n, compress
compression_rate, decisions questions/state); text, instructions, prompt,
context and model are rejected. An empty api_key no longer lets the
SCALEDOWN_API_KEY from the environment go to a custom api_base.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Read compress token usage from the real response shape
/compress/raw/ nests the per-prompt counts under results and reports the total
as a top-level input_tokens; the provider read a top-level original_prompt_tokens
that is not there, so compress reported zero tokens and zero cost.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Accept the router's max_retries and word the extra_body error clearly
The proxy adds max_retries to every call, which map_openai_params rejected, so
every proxied ScaleDown request failed with a 400. It is now accepted and not
forwarded.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Return extracted fields from extract, follow schema refs, and meet the type-discipline gate
- extract content is now the extracted fields (span anchors dropped, wrappers
unwrapped), so it validates against the caller's response_format; the raw
payload stays on _hidden_params['scaledown_response'].
- Local $ref/$defs in the response_format schema are resolved.
- The adapter is rewritten with Final locals, read-only annotations, one raising
helper and a typed handler; check_type_discipline reports no violations.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Fix stream_options, schema-guided extract cleaning, document input and multi-image handling
- stream_options is accepted and not sent upstream, so include_usage streams work.
- Extract output is cleaned against the requested entities, so a field named
_value or *_span_anchor is no longer dropped.
- Extract and summarize send an image or PDF as the native document fields.
- More than one image is rejected instead of silently using the first.
- The cost test derives its rate from the cost map.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cut recursive schema refs at the cycle and cap the expanded entity count
A small response_format schema with a self-referencing $def expanded to 4^16
leaves before any request was sent. Refs that point back into themselves are now
cut off at that point, and the total expanded entities are capped at 1000 so
repeated acyclic refs are rejected with a clear error.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Bound ref chains and schema nesting so long schemas fail cleanly
Resolving a chain of thousands of distinct $refs recursed until RecursionError.
Ref resolution is now an iterative walk capped at 32 hops, and nested properties
are capped at 32 levels; both raise a clear 400 instead.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(scaledown): support native classify and preserve extraction results
* test(scaledown): include provider regressions in CI shard
* fix(scaledown): pass registry typing and bounded recursion checks
* docs(scaledown): capture provider setup before and after
* fix(scaledown): preserve nullable schemas and reject malformed requests
* fix(scaledown): isolate provider dispatch from existing providers
* fix(scaledown): preserve the shared dispatch return contract
* fix(scaledown): preserve failure logging and reject lossy schemas
* fix(scaledown): keep timeout typing annotation on its expression
* fix(scaledown): resolve escaped extraction schema references
* fix(scaledown): annotate the bounded reference walk
* chore(scaledown): remove unused loop suppression
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: moe-berri <moe@berri.ai>
* refactor(anthropic): move passthrough SSE stream assembly into litellm/llms
Part of LIT-9423
* refactor(anthropic): move raw SSE stream assembly helpers out of proxy guardrails
Part of LIT-9423
* refactor(anthropic): give the anthropic_sse re-exports reasoned noqa codes
Part of LIT-9423
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* test(anthropic): cover the raw SSE helpers in stream_assembly
Part of LIT-9423
* refactor(anthropic): move prompt cache pricing helper out of the proxy package
Part of LIT-9423
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): move prompt cache observation lookup out of the proxy hook
Part of LIT-9423
* test(internal_context): exempt moved prompt cache lookup helper from key-family check
* perf(ci): distribute unit cases and reuse security sweep connections
* revert(ci): reject first speed trial after new failures
* perf(ci): avoid repeated test discovery and parsing
* perf(ci): isolate parallel security workers
* revert(ci): restore measured security execution isolation
* perf(tests): reuse source scans and deduplicate cache cleanup
* perf(ci): restore parallel security with safe log batching
* test(ci): isolate fixture state and schema comparisons
* test(ci): make schema and scheduler fixtures deterministic
* test(ci): drain queued logging before fixture teardown
* test(ci): preserve logging delivery without idle worker restarts
* test(ci): keep guardrail sinks available during parallel bursts
* test(ci): isolate full proxy lifecycle from gateway imports
* test(ci): preserve first failure responses and sweep evidence
* test(ci): restore documentation URL settings between cases
* test(ci): isolate open MCP tools in private gateways
* test(ci): group private MCP cases and handle exited workers
* test(ci): retain proxy exits and MCP failure details
* test(ci): repair merge interactions with main
Key the schema shard's full clone on artifact-name, since main removed the
unit-flag input. Give each owned Redis its own log file again. Hand the
realtime transcription tests main added a logging object whose dispatch
handlers are awaitable, so the per-test logging drain no longer waits on a
MagicMock task forever
* perf(ci): shard owned integration cases and close lifecycle leaks
* fix(ci): qualify shard execution and harden interrupted cleanup
* fix(ci): bound worker supervision and own fault traffic
* perf(ci): reuse unparametrized fixtures across bounded case groups
* fix(ci): preserve boot fault allowances and complete exit records
* test(ci): tolerate startup retries and retain readiness details
* perf(ci): balance integration shards by recorded duration
* fix(ci): load migration database helpers only for workers
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream
* test(websearch_interception): type the conversion helpers and drive the replay test through mock_response
* test(websearch_interception): pin the replay test's values as Final and record follow-ups immutably
* fix(proxy): strip every agentic-loop control key from client bodies, the web search keys included
* fix(interception): move stream_options off forced non-stream calls for code interpreter and headroom, and keep adapters from replaying nested calls
* test: annotate new interception test locals with Final
* fix(responses): drop params a deployment hook removed on sync Responses calls
* test: drive the sync Responses stream_options regression through a mock transport and real Azure config
* test: type the agentic loop recorder and capture the sent request on an object
* test(integration): cover interception stream_options on chat, responses, messages and the SDK
* test(integration): assert the follow-up error text and release a held burst on failure
* test(integration): pin the follow-up failure contract and wait for the replacement worker before teardown
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility
* fix: preserve HTTPX compatibility across OpenAI SDK versions
* refactor: trim OpenAI SDK compatibility patch
* refactor: simplify SDK client defaults
* test: cover SDK compatibility across provider routes
* fix: use shared names for SDK transport helpers
* fix: address SDK compatibility review feedback
* test(openai): send requests through the SDK API factory clients
* fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2
* ci(base-sdk): count httpx2 as base-only when a base dependency brings it in
OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure
* test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively
* test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10
---------
Co-authored-by: Marcus Wood <marcuswood@openai.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices
* test(gemini): type the voice mapping test helpers
* fix(gemini): leave nova out of the voice mapping since Gemini accepts it as given
* test(gemini): add integration cells for the OpenAI voice mapping on Gemini and Vertex TTS
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): keep #/$defs refs in json_schema response_format for non-Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): type the json_schema response_format override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): integration cells for json_schema $defs refs across endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): split stacked loop in streaming json_schema cell
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): mark BaseConfig dict contract on json_schema override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): cover json_schema refs on every endpoint, edge refs, sad paths and an outage burst
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): keep GPT tool-result images beside the tool result
Bedrock gpt-6.1-sol rejects an image nested in toolResult.content. Put it on the same user turn instead.
* fix(bedrock): read tool-result image support from the price map
OpenAI GPT rows on Bedrock Converse reject an image inside toolResult.content, so those images sit beside the tool result.
* fix: sync price-map test schema and import order
CI validates model_prices_and_context_window.json against an inline schema in test_utils, and ruff checks import order in converse_transformation.
* fix(bedrock): satisfy basedpyright in tool-result image helper
Drop Final assignments inside loops and use TypedDict key checks so the lint type-check gate stays within its merge-base ceiling.
* test(bedrock): cover tool-result image placement helper
Exercise Claude no-op paths, text-only GPT tool results, status preservation, and split text/image tool content for codecov.
* refactor(bedrock): reuse the shared tool-result image placeholder text
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider
* fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/
* fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor
* fix(ui): name every OSS classifier provider in the classifier radio copy
* feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry
* fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry
* fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router
A /model/new for an auto router lands on one replica. A request routed to
that auto router on a sibling replica missed it, the model read-through
loaded only the auto router's own row, and the pre-routing hook then picked
a tier the sibling had not loaded, so the caller got a 400 "no healthy
deployments for <tier>" until the next periodic reload. An auto router miss
now runs the full add_deployment reconcile, tiers included. Found by the
audit's two-worker rig on this PR's routing cells; CircleCI's one-worker
proxy never opens the window.
* test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier
Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing
cells for the complexity router's databricks classifier on all three
endpoints with the Postgres rows read back, management cells for the
classifier's spend writes and for the evaluation-mode health check of a
Databricks serving endpoint.
* chore(github_copilot): sync model catalog with Copilot's served models
* test(github_copilot): assert catalog invariants instead of pinning vendor facts
* fix(github_copilot): keep thinking capabilities on served Claude catalog rows
* fix(github_copilot): keep supports_reasoning on catalog rows that shadow the family fallback
* fix(model-catalog): preserve merged Copilot Haiku metadata
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange
* fix(ui): create a credential for credential-only auth types in Add Model
* fix(microsoft_365_copilot): accept max_tokens and list the chat model
* fix(microsoft_365_copilot): register provider model set
* fix(ui): hide other auth types' fields when credential-only types are filtered
* fix(health): do not inherit stored auth settings when a connection test brings its own api_key
* docs(health): restore connection-test inheritance docstrings
* fix(lint): suppress justified provider boundary casts
* fix(lint): satisfy M365 type-discipline gate
* fix(types): eliminate type-check gate regressions
* style: shorten suppression reasons so ruff format leaves them on one line
* refactor(types): drop shared-file type widening unrelated to the Copilot provider
* fix(health): pass caller headers as secret_fields to connection-test probes
* fix(types): type connection-test probe and token counter locally
* fix(lint): drop cast import from connection-test header getter
* chore: remove explanatory comments flagged by review
* fix(proxy): gate OAuth client credentials behind proxy admin
* fix(router): skip cooldown on caller-scoped OAuth auth failures
* fix(router): skip fallback cooldown on caller-scoped OAuth auth failures
* fix(router): type caller-scoped cooldown lookups
* fix(ci): include Microsoft 365 Copilot tests in unit shard
* fix(proxy): keep saved endpoint when test_connection retests a deployment by id
* fix(m365): collapse doubled Graph replies
* fix(m365): preserve trailing system messages
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on
* fix(bedrock): sign the listing query RFC 3986 style and raise BedrockError on a failed listing
* test(utils): keep test_utils.py at main's formatting, only the Bedrock discovery test is new
* fix(bedrock): chain BedrockModelInfo's constructor so the passthrough config keeps its event parser
* fix(bedrock): list discovered ids without the provider prefix so partial wildcards filter
get_valid_models returns bare vendor ids for every other provider and for the static bedrock catalog, and the proxy's wildcard expansion adds the provider prefix itself. The lister prefixed its ids, so a partial wildcard like bedrock/anthropic.* never matched the proxy's filter and listed uncallable bedrock/anthropic.bedrock/<id> entries
* test(bedrock): integration cells for wildcard discovery through the proxy and the SDK
* fix(proxy): keep a partial wildcard a filter when its deployment repeats the prefix
The wildcard expansion guessed filter-or-alias by whether any provider id carried the prefix. A deployment such as model_name bedrock/anthropic.* over model bedrock/anthropic.* can only ever route names that share the prefix, so when the account has no on-demand anthropic.* id (every Anthropic model behind an inference profile) the guess fell into the alias branch and listed bedrock/anthropic.<every id>, none of them callable. A deployment whose model repeats the suffix now always filters, and lists nothing when nothing matches
* fix(proxy): prefix every discovered id under a custom wildcard prefix that starts a vendor id
* test(bedrock): Final and read-only annotations in the wildcard discovery tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): route Grok 4.7 tools with reasoning to native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): mint unique tool call ids on native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send a blank text block for Converse tool results with no supported content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): default Grok on Bedrock to native Chat Completions and leave Converse unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): gate tool call id minting to Grok on native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): check the Grok tool call id gate at both call sites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): remint positional tool call ids on native Chat Completions for every model
AWS returned call_0, call_1 for GPT 5.6 as well as Grok on 2026-10-09 (and unique ids for GPT 5.6 earlier the same day), so the id format is not fixed per model. Drop the Grok-only gate and remint any call_<digits> id on the route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop stop sequences for Grok instead of forwarding them to a 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): poll the partition lock witness off the event loop
The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting
* test(e2e): retry the reasoning turn until the model emits a reasoning item
Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion
* test(e2e): retry a Bedrock Converse model once when it reports no cache tokens
Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged
* test(e2e): poll until a deleted stored response stops being retrievable
Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx
* test(e2e): give the Together prompt cache five primed attempts
Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed
* test(e2e): allow 30s for the idle RSS reading at session start
A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged
* test(e2e): scope the MCP submission check to its own card
The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card
* test(router): wait for the primary success record before counting it
The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs
* test(proxy): give the spend-log cleanup run a 1s budget
A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves
* test(autorouter): release the slow token count at 2.05s instead of 2.3s
The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack
* test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning
grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting
* test(integration): give the two worker-kill chaos tests the owned-proxy time budget
Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests
* test(integration): compare regional image responses without their created timestamp
Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared
* test(integration): count only the S3 logger's flush task
The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels
* test(integration): send guardrail timeout probes eight at a time
All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits
* test(integration): count a killed Arize chaos worker as gone once it is a zombie
psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget
* test(integration): send the Typesafe connection test its real model and endpoint
The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly
* test(integration): keep DB-stored models out of the usage-routing Redis read test
The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off
* test(vertex_ai): prove the batch upload streams by laziness instead of peak memory
The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests
* test(integration): keep the alias test_connection call as a known bug
* test: type the counting mapper and the Bedrock rerun helper
* test: annotate the new test locals as Final and build them as tuples
* test(vertex_ai): prove pull-driven transforms with the garbage tail alone
The two offline tests in tests/local_testing/test_anthropic_prompt_caching.py failed on main because
#24071 started passing logging_obj to client.post, so assert_called_with on the patched
AsyncHTTPHandler.post no longer matched. The request bodies themselves were still correct
Both now live in tests/unit/llms/anthropic/chat and assert the body and headers that reach the
wire through respx instead of patching our own HTTP handler. The coverage allowlist entry keeps the
five tests that still need provider credentials
Moves 58 legacy nodes (cyberark, logging callback manager, langfuse handler and helpers, SQS, app crypto, bedrock nova and invoke, cloudflare, litellm_proxy, nvidia nim, replicate, triton, convert_dict_to_response, get_model_info, vcr live-call probe) into the tests/unit files that mirror the modules they exercise. Calls that reached huggingface.co, langfuse or 0.0.0.0 are refused at the HTTP boundary with respx. Tests that need credentials, a license, a spawned server, sleeps, a subprocess, or assert nothing stay in their legacy files
* fix(anthropic): count a leading system run through count_tokens' system parameter
Anthropic's count_tokens rejects role "system" at the head of messages, so
/v1/responses/input_tokens with instructions, and /v1/messages/count_tokens
with a system-role message, fell back to the local tokenizer. The shared
Anthropic count_tokens transformation now lifts the leading run of system
messages into the top-level system parameter, the way the chat path sends
it, after any system the caller set. Anthropic direct, Azure AI Anthropic,
and Bedrock Mantle share that transformation.
* test(integration): cover the count_tokens leading-system lift across Anthropic, Azure AI and Bedrock Mantle
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): split the <reasoning> tag for gpt-oss only on native Chat Completions
The native Chat Completions route moved a leading <reasoning>...</reasoning>
block into reasoning_content for every model, while only gpt-oss writes its
reasoning inline. A GPT 5.6 or Grok answer that starts with a literal
<reasoning> tag lost that text, streaming and non-streaming alike. Both paths
now split only when the model id is gpt-oss.
* fix(bedrock): drive the inline reasoning split from a cost-map flag
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
#45214 routed both decision formats through the shared decisions IR in
litellm/llms/base_llm/decisions/transformation.py and litellm/types/decisions.py.
That left litellm/llms/base_llm/decisions/systemone.py (to_system_one_request,
question_keys, to_decisions_response, the SystemOne* models and their adapter)
and litellm/types/openai_decisions.py with no importer outside their own unit
tests, so both modules and both test files go.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): run each mode only on streaming, non-streaming, or both
Add stream_scope so a rail can target streaming inference, non-streaming inference, or both per pre, during, and post mode
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor stream_scope in pipelines and dashboard types
Pipeline steps skipped the stream_scope filter, direct construction ignored mixed-case maps, and schema.d.ts was stale.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): skip unmatched stream_scope steps instead of allowing
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): keep stored stream_scope keys for modes not on screen
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): format stream_scope helpers and update fork MCP unit tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): keep hang cancellation tests from timing out during setup
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor path-defined streaming for stream_scope
Passthrough routes like Gemini streamGenerateContent decide streaming from the URL, so stamp that onto hook data before guardrails run.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): copy AnthropicModelCapabilities instead of cloning
Clippy treats clone-on-Copy as an error, which failed rust-lint on the messages request tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): trust only server stream classification
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): keep streaming marker through deepcopy
scan_raw_request snapshots copy each field, so a plain object() marker would lose identity and skip streaming-only rails.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cost-map): drop duplicate perceptron-mk1.5 row
Two main cost-map PRs both added the OpenRouter model, so the merge left a second key that CI rejects.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): expect native transcription 429 as RustUpstreamError
HTTP status errors from native routes map through route_error_to_pyerr, so the wheel SIGINT child was dying on an outdated RuntimeError check and never reached the hang probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): follow Google Interactions OpenAPI without hardcoded names
The live spec dropped CreateModelInteractionParams and renamed the item path to {interactionsId}. Misc CI failed because the compliance tests still looked those names up as literals.
* test(guardrails): reproduce stream_scope bedrock and passthrough field gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover invalid stored stream_scope reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify bedrock stream actions and keep caller is_streaming_request
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): make stream_scope_allows public and drop mutable builds
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify pass-through and Bedrock stream scopes
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden stream classification and validation
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add stream scope integration audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid mutating passthrough custom body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): run stream scope audit without enterprise license
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): keep stored scope restart cell on one worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): scope stream scope audit sink assertions to the rail under test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert logging_only scope absence behind an ordered barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert one logging_only scan per phase after the barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): set request scope on pass-through stream fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid YAML stream_scope in v1 guardrails list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: merge main into litellm_guardrail_stream_scope_fixes
Update pass-through pre-call test callbacks for main's endpoint_type argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add request paths to pass-through fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover websocket pass-through stream scope and LIT-9050 outage spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): remove unused type discipline suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): script the vertex live upstream in the websocket stream scope test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep stream marker json-serializable and restore pass-through helper names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): allow required Bedrock action re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): strip the stream marker from pass-through payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): drop stream marker by value in scans and snapshots, plain-tuple stream scope state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): render tag-scoped guardrail modes read-only in the custom code editor
A guardrail whose litellm_params.mode is the tag-scoped dict {tags, default}
crashed the Custom Code editor on open: normalizeMode wrapped the dict into
the mode array and StreamScopeFields rendered it as a React child (error #31,
whole dashboard unmounted). Treat a non-string non-array mode as no editable
modes, show formatGuardrailMode(mode) in a disabled input (read-only, matching
the guardrail info view), and keep mode/stream_scope out of the update payload
for such guardrails.
* fix(ui): resolve merge fallout in guardrails components
Deduplicate toModeArray import after the merge, and move the read-only
guardrail details block back into GuardrailReadOnlyDetails (now rendering
the shared mode/logging-only rows plus the stream-scope detail) so
guardrail_info.tsx stays under the 800-line lint budget. Guardrails UI
suite: 298 passed.
* fix(ui): drop duplicate toModeArray import reintroduced by merge
* chore(pass-through): document the deliberate in-place marker strip as mutable-ok
The clear/update on _parsed_body is load-bearing: rebinding to a fresh
mapping instead breaks 75 pass-through tests because the marker-free body
must propagate through the caller's request dict so downstream guardrail
scans and snapshots never observe the server streaming marker.
* fix(guardrails): move stream_scope after timeout in CustomGuardrail init
Inserting stream_scope before the existing timeout parameter shifted the
positional slot of timeout, so positional callers constructed with their
timeout bound to stream_scope (ValueError) and timeout silently None.
Restores the base parameter order; keyword callers are unaffected.
* fix(guardrails): typing pass for the lint gates
stream_scope leaves the declared constructor parameters (restoring the
base positional surface; keyword construction unchanged), unknown config
values crossing the new stream-scope code paths get typed locals or
cast-ok boundaries, and the passthrough payload literals are annotated.
All three lint gates pass against current main; the guardrail suites are
unchanged (236+5019 passing; one known anyio-driver failure pre-existing
on base).
* style: sort cast imports for the ruff gate
* chore(pass-through): drop unused BEDROCK_STREAMING_ACTIONS re-export
The streaming check now uses is_bedrock_streaming_endpoint; nothing in
the repo imports the name from this module.
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: gabriele <gabriele@berri.ai>
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings
* test(gemini): annotate the embedding mock helper and keep the type comment short
* fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly
* test(caching): type the partial-hit cache test and pin its clock
* fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id
* fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs
A bare object `input` answered 500 from both the caching handler and the
transformation; both now answer 400. An unresolved `files/` reference on
vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/
provider. An empty `format` passed through to the provider; it now answers 400
naming file.format. Unknown block keys (`detail`, an unknown video_metadata
key, a top-level block key) are dropped under drop_params, global or
per-request, and still answer 400 without it. The embedding cache counted
file blocks against `max_messages`, so a request with 5 or more blocks was
never cached; file blocks no longer count.
* fix(caching): answer 400 for an object embedding input on the cache lookup
* test(gemini-embeddings): annotate the new embedding tests with return types and Final
* test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final
* test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos
Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends
* test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(voyage): rebrand to VoyageAI by MongoDB, refresh model list, support both contextual input shapes
* fix(voyage): send auto-chunking params for flat contextual inputs
A flat list[str] or bare str for the contextualized embeddings endpoint is
only valid as documents with enable_auto_chunking=True and input_type=document,
or as queries with input_type=query. Default those params for non-query flat
inputs so the request matches the live API contract, while letting caller-set
values win. Nested list[list[str]] still passes through unchanged.
Tests now assert the auto-chunking params instead of only echoing inputs.
* feat(voyage): route MongoDB-issued keys to ai.mongodb.com
Mirror voyageai.util.get_default_base_url from the official SDK: a key with
the `al-` prefix is issued by MongoDB and is only valid on ai.mongodb.com,
every other key on api.voyageai.com. The choice now lives in one shared
helper that the embedding, contextual, multimodal, and rerank configs all
use for both the default base URL and the Authorization header, so the host
and the key always agree.
Also drops the voyage-4-nano and voyage-multilingual-2 model map entries:
voyage-4-nano is not served on the Voyage API and voyage-multilingual-2 is
an older model, so neither belongs in this change.
* fix(voyage): restore MongoDB routing on contextual endpoint and re-add voyage-4-nano after upstream merge
The upstream merge replaced the contextual config's get_complete_url with a
Voyage-only host and dropped voyage-4-nano from the model map. Reapply the
al- key routing via get_default_base_url and add voyage-4-nano (open-weight,
per docs.voyageai.com) so the branch keeps its task changes on top of upstream.
* fix(voyage): drop voyage-4-nano and the contextual tests upstream already covers
voyage-4-nano is not served on the Voyage API, so it does not belong in the
model map. The contextual input tests duplicate
tests/test_litellm/llms/voyage/test_voyage_contextual_embedding.py, which
landed upstream with the auto-chunking fix this branch was carrying.
* test(voyage): drop sys.path.insert from the voyage common utils test
* fix(voyage): route rerank on the key it authenticates with
get_complete_url picked the rerank host from the environment while the auth
header carried the request key, so an explicit MongoDB-issued key was posted
to api.voyageai.com. validate_environment now hands the resolved key to the
config instance and get_complete_url reads it back, so the host and the
credential always come from one key. The config is built per request, so
nothing carries over between them.
* test: allow ultrafast pricing keys in model prices schema
* test: drop ultrafast schema keys now added upstream
* test(integration): prove voyage key-based host routing on the wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(fireworks_ai): forward the LiteLLM user id as user behind fireworks_forward_user_id
* fix(fireworks_ai): keep the forwarded user id when the caller sends extra_body.user
