* fix(e2e): route credential, cost map, and UI login calls to the control plane
* test(e2e): stop pinning the upstream model name in the credential-backed messages test
* test(e2e): drop the harness unit test, the split stack run proves the routing live
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): opt-in include_guardrail_response returns guardrail_information in the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read include_guardrail_response from the request metadata bucket the router did not reseed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format common request processing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): redact matched content in guardrail_information and stop mutating cached responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): traverse guardrail diagnostics iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): annotate guardrail traversal cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse core redaction helper for guardrail_information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): justify response rebind when attaching guardrail information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval
* test(e2e): skip the sibling replica cooldown cell when one gateway URL is named
* test(e2e): collapse repeated gateway addresses so the sibling cooldown cell skips instead of erroring
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): propagate parent session/trace ids into intercepted searches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch_interception): let parent correlation win over configured search params and type test params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bill an intercepted web search under the parent request session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop unrelated reformatting from the websearch session harness change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the websearch interception session suite out of the stage-mirror gate
The stage-mirror stack runs no websearch_interception callback or search tool, so the suite is deselected there and the changed-tests gate fails on a file that executed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): run the websearch interception session suite on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): leave CONTRIBUTING.md untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langsmith): json.dumps with default=str so non-serializable metadata does not crash batch flush
Serialize the runs/batch payload with json.dumps(default=str, allow_nan=False) and send it as content= with an explicit Content-Type, so datetime, Decimal and similar metadata values no longer raise TypeError and drop the batch. Forward content= on the AsyncHTTPHandler retry path so a retried batch re-sends the identical body
Replaces #39133, which was cut from the retired staging branch and conflicts with main
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): drop test docstrings and replace monkeypatch with a client-injecting handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): add live e2e for non-native metadata reaching LangSmith
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): scope the e2e docstring to the values the test injects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(http_handler): close injected retry clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): deselect the LangSmith live e2e on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): write key deleted audit logs for cascade and alias key deletions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert persisted key deleted audit rows for cascade paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): route /audit and /v2/login to the control plane in split transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): surface an upstream handshake refusal as an error event and policy close
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): tidy the handshake refusal e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): keep upstream exception text out of the Azure client error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(realtime): map handshake refusal close codes with a lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): map rerank and search output and the OCR, image edit and search input onto the Langfuse generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): summarize OCR data URIs by media type and size and log an empty document URL as empty
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): keep URL-less search results, name OCR file streams and skip non-str query parts when logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel v2): drop the unused typing imports and the decorative section divider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): accept a team alias in x-litellm-team-id
The header only matched canonical team ids, so a JWT caller selecting one of their teams by its alias got a 403 even though they belonged to it. The header value is now resolved through the existing alias lookup before the JWT allowed-team check and the DB membership fallback, while a value that is already a team id never costs an alias lookup and denials keep naming the value the caller sent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): only alias a header team id the database provably lacks
Under fallback_to_db_teams a header value whose team row read fails for any reason other than TeamNotFoundError now keeps the membership denial instead of falling through to the alias lookup, so a degraded read cannot select a different team that carries the value as an alias. Drops the HeaderTeam docstring that only restated its fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): store the masked output in spend logs when a post_call guardrail rewrites the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): record served output without re-narrowing the logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): overlay the served output before message redaction so turn_off_message_logging still wins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): type the monkeypatch fixture in the redaction ordering regression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll through raw card output until the guardrail reaches the serving worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep blanked, multi-choice and disconnected served output out of raw spend logs
Served text keeps empty strings and tracks unavailable choices as None so a guardrail that blanks
the output still overrides the raw provider text. Stream choices are sized from the highest choice
index, served chunks are recorded before a client disconnect or stream failure propagates, and
message-logging redaction drops the served text from callback kwargs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): add conversational matrix across chat, messages and responses
Parameterizes one behavioral contract (reply, stream, cost log, tool call,
tool round trip) across /v1/chat/completions, /v1/messages and /v1/responses,
OpenAI and Anthropic models, and env-ref vs stored-credential auth, with
record/replay fixtures.
Adds general_settings.disable_model_info_refresh so the proxy fronting a
replay fixture does not poll every OpenAI-compatible deployment's /v1/models
in the background, which otherwise leaves unconsumed interactions in the
recorded bundle.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): force the weather tool on the first turn and rename Provider to Deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): map completions, images, speech, transcription and moderation output onto the generation output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redaction): redact text completion choices in the standard logging payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): compare decoded generation output text and follow the live moderation verdict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel v2): compare logged byte counts with the received media and move e2e schemas into models.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the otel_v2 Langfuse output e2e file out of the stage-mirror gate it cannot run in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): mask PII in streaming /v1/messages output
Raw Anthropic SSE frames were passed through the post_call output masking
callback untouched, and ProxyLogging rerouted the callback to the unified
apply_guardrail path on /v1/messages because mask_response_content was
false. Buffer the raw frames, assemble them with the shared Anthropic SSE
helpers, mask through Presidio, and re-emit the masked frames.
Resolves LIT-8288
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): replay raw SSE frames when masking fails mid-stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): propagate upstream stream errors instead of returning an empty stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): extract buffered stream masking to satisfy complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): let BLOCK on generated PII refuse the streaming /v1/messages response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): fold the BLOCK re-raise into the existing except to stay within the complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): move the blocked stream consumption into a helper so pytest.raises holds one statement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): fail closed when output masking of a raw SSE stream errors
A Presidio outage on streaming /v1/messages replayed the unscanned frames
to the caller. Propagate the error instead, matching the non streaming
path and the merge base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): cover structured chat stream output masking and trailing bytes passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e-ui): check the MCP Tools tab against the upstream's own tools/list
DeepWiki renamed ask_question to ask_wiki_question, and the spec hardcoded the old name, so
e2e_ui_testing went red on main for something that is not a litellm regression. The spec now asks
the upstream server for its tool list with the official MCP TypeScript SDK and expects the tab to
show exactly those cards, so a vendor rename cannot turn the job red again.
* test(e2e-ui): cite the pinned DeepWiki tool name and drop the helper docstring
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep S3 signer test additions scoped to new cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): drop e2e suite changes from the S3 signing fix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): build S3 credentials directly from the s3_* pair so ambient AWS_* env never mixes in
Restores the split-identity e2e coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan video prompts for key-attached guardrails on /v1/videos
/v1/videos dispatches call_type avideo_generation, which CallTypes did not
know and no guardrail translation handler covered, so the unified guardrail
hook returned the request unscanned. Add the video call types and an OpenAI
video guardrail translation package that scans the prompt for create, remix,
edit and extension requests
Resolves LIT-6685
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api types for video call types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: skip avideo_generation in azure sdk client exhaustive check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): retry a leaked video job until the guardrail sync deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): satisfy the type-discipline gate in the video handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): gate the video e2e on a chat probe so a miss starts at most one paid job
Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolve conflicts in the e2e Responses suite: main replaced the bespoke
EndpointsClient with provider SDK clients (76bf0cd579), so the four added
Vertex and Azure OpenAI /v1/responses tests are ported onto the OpenAI SDK
and tool_choice=required now comes from the SDK instead of a custom
request model.
Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/v1/videos dispatches call_type avideo_generation, which CallTypes did not
know and no guardrail translation handler covered, so the unified guardrail
hook returned the request unscanned. Add the video call types and an OpenAI
video guardrail translation package that scans the prompt for create, remix,
edit and extension requests
Resolves LIT-6685
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Register s3_access_key_id, s3_secret_access_key and s3_encryption_key_id as
LiteLLM-owned batch params so they are no longer forwarded to Bedrock as
additionalModelRequestFields (which 400s ordinary chat on a batch-configured
deployment), keep them on CredentialLiteLLMParams so the batch/file paths
still receive them, and redact the S3 credential key names in debug logs.
Resolves LIT-8290
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Register s3_access_key_id, s3_secret_access_key and s3_encryption_key_id as
LiteLLM-owned batch params so they are no longer forwarded to Bedrock as
additionalModelRequestFields (which 400s ordinary chat on a batch-configured
deployment), keep them on CredentialLiteLLMParams so the batch/file paths
still receive them, and redact the S3 credential key names in debug logs.
Resolves LIT-8290
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Three holes in the new suite, all of which let a test pass without proving
what its name claims:
- A migration recorded twice, once per replica, each with
applied_steps_count = 1, slipped past both the step-count check and
migration_names(), which collapses the history into a set. Reject
duplicate migration_name rows outright.
- auth_traffic only asserted the failures it had seen by the time
keep_serving hit its target. A request failing after that, or on the
other replica while the test waited on one stream, was recorded and
never read. Assert the recorded failures once the thread has joined.
- The rolling test warmed the baseline replica's virtual-key cache before
the upgrade, and that cache holds for 60 seconds by default
(UserAPIKeyCacheTTLEnum.in_memory_cache_ttl). The candidate migrates
well inside that window, so the post-upgrade requests could be served
from cache without ever repeating the whole-row token lookup that the
stale prepared statement breaks. Drive the baseline replica with a key
minted after the schema moved, which it has never seen and must resolve
from the database.
Re-ran against v1.101.0 -> v1.102.0: 6 passed.
The migration e2e harness only ever used one image: it seeded the database
with the candidate build and then applied synthetic migrations on top. That
proves the migration machinery (locking, crash recovery, legacy baselining,
pooling) but never executes the real schema of release N against the real
migrations of release N+1, which is the path operators actually run.
Adds a baseline image alongside the candidate, so a test can seed with a
published release and upgrade with the build under test.
Suites:
- test_upgrade.py: the candidate applies the pending release migrations,
keys minted by the baseline release survive, and concurrent replicas
upgrade a baseline database exactly once.
- test_rolling_upgrade.py: a baseline replica keeps serving virtual-key
auth while the candidate migrates underneath it, and both releases serve
and resolve each other's keys during the overlap. This is the reported
failure: a new column on LiteLLM_VerificationToken invalidates prepared
plans on pods still running the old release, which the proxy reads
whole-row, and auth starts failing until those pods leave service.
- test_shaped_database.py: the upgrade completes and preserves rows on a
populated spend log, rather than on the empty database every other
migration test starts from.
Every upgrade assertion is gated on the candidate having actually applied
migrations the baseline had not, so a stale pin fails loudly instead of
passing on an empty delta.
CI adds two jobs to the migration_startup workflow. The baseline defaults
to a committed release pin and is overridable per pipeline, matching how
migration_candidate_image already works; only the upgrade jobs pull it.
Verified against a real v1.101.0 -> v1.102.0 upgrade: 6 passed, with the
baseline seeding 165 migrations and the candidate applying the 6 that
landed between the two releases.
The deleted wrapper put cache: {"no-cache": true} on every request body, so the
gateway's response cache never answered a re-sent prompt. The SDKs send nothing
of the sort, and the mid-conversation prompt-cache priming loop re-sends an
identical body until the provider reports a warm cache, which a cached reply
never does. NO_PROXY_CACHE in sdk_clients.py restores the field as extra_body
on every messages, responses, completions, and embeddings call.
The wrapper also returned a 4xx as a value where the SDKs raise. The Bedrock
safety_identifier test judges the captured Converse body, and Claude on Bedrock
rejects the forwarded field with a 400, so the /v1/responses leg now suppresses
openai.BadRequestError the way the chat leg carries the same 400 as a Result.
Live proxy with cancel_on_disconnect, two-deployment group, generic allowed_fails=0, red at the pre-fix handler and green with the bare raise
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolves the six conflicts main accumulated against the SDK migration.
endpoints_client.py stays deleted (main only added the safety_identifier
kwarg to it), and its two new importers move onto the SDK clients: the
Bedrock safety_identifier forwarding test in test_responses_e2e.py now
sends its /v1/responses leg through the OpenAI SDK and its chat leg
through ProxyClient.chat, and test_messages_azure_foundry_e2e.py drops
the x-litellm-rust header check together with EXPECT_RUST, which main
removed from e2e_config in a84f68b6e3. proxy_client.py keeps both the
rerank and router-settings model imports. uv.lock is main's lock plus
the anthropic e2e-dev entry, and the pricing JSONs are taken verbatim
from main since this PR never meant to change them.
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The changed-test gate booted its stage-mirror stack without files_settings
or finetune_settings, so every raw upload with a custom_llm_provider hit a
500, and it exported the whole provider env into the gateways, so the
AWS_ROLE_NAME the assume-role test needs made the GovCloud deployment run
an AssumeRole with its static keys. The gate also deleted its pytest output,
so a red run left nothing to read. The mirror config now carries the
openai, azure, and vertex_ai file settings, gateways start without
AWS_ROLE_NAME, and the workflow uploads the pass logs and junit files with
every secret value, every field of a JSON-valued secret, and their
XML-escaped forms replaced before the raw files are removed.
