* test(e2e): failed request error span carries the full untruncated message and status
Covers logging.otel.failure.exports_metric on chat_completions: a request that
fails at the provider (invalid upstream key deployment) must export one
complete trace whose gen-AI span carries the LIT-4179 error contract, declared
as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated
error.message proven by parsing the embedded provider error JSON back out of
the attribute. The root SERVER span must record the 401 the client received.
Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which
the suite's model-registering tests already assume
* test(e2e): clean failure diagnostics on the error-span contract per review
A truncated error.message with missing braces now fails with a readable
assertion instead of an unhandled ValueError, an unparseable embedded JSON
fails via pytest.fail with the truncation context, and the retry loop now
asserts the upstream provider failure was actually observed so a fresh-key
propagation deadline cannot masquerade as a trace-export failure
* test(e2e): pin the full error attribute set including the litellm.provider.error keys
The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message
* test(e2e): author the error-span test docstring
* test(claude_code): move the Claude Code compatibility matrix under tests/e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: restore the upload-coverage job dropped by mistake with the compat gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments
* test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The rate-limited batch spend test snapshotted unattributed rows via the
unpaginated /spend/logs whole-table read, which grows with the environment
(58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every
scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an
explicit date window instead, and SpendLogsParams now rejects a filterless
read so the whole-table call cannot come back
* test(e2e): OTEL trace completeness on /v1/messages
Extends the LIT-3787 trace-completeness suite to the Anthropic-native route:
one successful non-streaming /v1/messages call must land at the destination as
ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT
span, no dangling parents). Adds the raw /v1/messages sender to the logging
suite client.
* test(e2e): reuse the shared AnthropicMessagesBody per review
Drops the duplicate /v1/messages request model in favor of the one models.py
already provides (budget_client uses the same one), passes max_tokens at the
call site to match the sibling chat test, notes in the docstring why the
gen-AI span is named chat on this surface, and adopts the hardened read-back
signature
* test(e2e): author the messages trace test docstring
* test(e2e): declare the messages surface on the covers marker
* test(e2e): otel trace completeness on /v1/responses (#33134)
* test(e2e): OTEL trace completeness on /v1/responses
Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API
route: one successful non-streaming /v1/responses call must land at the
destination as ONE connected trace. Adds the raw /v1/responses sender, a
CHEAP_OPENAI_MODEL config constant, and registers responses in the otel
registry cell's exercised_on.
* test(e2e): author the responses trace test docstring
* test(e2e): declare the responses and chat surfaces on the covers markers
* test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination
Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage:
a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix
preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export
through a preset-owned provider - the code path where trace splits happen), a
typed Jaeger query read-back client, and the first test: one successful
non-streaming /chat/completions call exports ONE complete trace (root SERVER
span + auth/db/cost children + gen-AI CLIENT span, no dangling parents).
* test(e2e): harden the otel trace read-back per review
Jaeger reads now query server-side by the litellm.call_id span tag instead of
paging recent traces and filtering client-side; the compose stack's background
jobs alone can push a request trace past the page. A failed query hard-fails
instead of reading as an empty result, the settle predicate now also waits for
the prefix-matched db span the assertion demands, parent-chain walking follows
CHILD_OF references only, the zero-trace and split-trace failures get distinct
messages, jaeger gets a healthcheck so the depends_on condition is accurate,
and the chat docstring names the route the code actually asserts
* test(e2e): author the chat trace test docstring
* Update logging section in CLAUDE.md
Removed mention of OTEL trace-tree completeness from logging integration section.
* fix(guardrails): filter Add-Guardrail mode dropdown per provider
The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.
Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.
Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).
Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.
Resolves LIT-4226
* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form
Address Greptile P1 (startup break) and P2 (edit form UX):
LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.
Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.
* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint
Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.
Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.
Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions
Exclude vertex_ai from pipecat tool smoke; raw-ws tool_call_round_trip
remains the Vertex source of truth. Also remove the Playwright key models
dropdown suite so stage is not blocked by that UI harness
* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells
Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat
traffic and assert calculatedTotalCost matches StandardLogging response_cost
and proxy spend. Also assert tool calls and applied guardrails land on the
trace. Missing env or proxy is a hard failure, never a skip
* test(e2e): use langfuse_otel callback for Langfuse spend coverage
Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to
Langfuse) instead of the classic langfuse SDK. Match generations named
litellm_request by prompt marker and user_api_key_alias
* test(e2e): require Langfuse spend assert; drop AGENTS.md
Guardrail path no longer soft-gates logs_spend. Non-stream responses must
return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md
* test(e2e): fail when Langfuse spend is missing on guardrail path
Always run logs_spend assertions for tool_permission; require positive
x-litellm-response-cost on non-stream and positive /spend/logs spend
* test(e2e): do not fall back to unmatched spend log rows
poll_proxy_spend_for_key returns None when response_id or positive-spend
filters match nothing, instead of silently using rows[0]
* fix(e2e): wire batch provider secrets for docker and k8s
Point batch deployments at the credential field names and os.environ refs
the gateway actually resolves from process env (compose .env or EKS secret
mounts). Missing secrets skip instead of failing red so a red run means a
product bug. Mirror S3 bucket env aliases in docker-compose for provider_fallback
* fix(e2e): drop batch provider_env unit tests
The batches suite is live e2e only; no monkeypatch or unit-level tests
* fix: batch credentials, provider list, and team db lookup
Keep object-storage fields through CredentialLiteLLMParams and resolve
os.environ/ refs when reading deployment credentials so Vertex/Bedrock
batch file uploads see bucket and AWS keys from K8s/docker env
Skip managed batch list when the request is provider-scoped so
/{provider}/v1/batches list works instead of 500
Force DB on check_db_only team lookups and stop masking non-404 errors
as "team doesn't exist"
Drop e2e runner-side skip helpers; hard-fail on missing gateway secrets
* fix: tag reseed, team window spend, and remaining e2e flakes
Reseed spend:tag counters from LiteLLM_TagTable so cold redis still
enforces after the spend writer flushes
When applying post-call cost to team multi-window counters, load the
team from the DB if it is missing from the management cache so window
spend is not dropped on cache misses
Harden cold-counter reseed e2e (namespace-aware keys, burst success,
poll). Give tag budget more headroom. Retry /key/update on redis DNS
blips. Ensure NLTK punkt_tab is present for pipecat realtime audio
* revert: drop product code changes; e2e-only scope
Reverts all litellm/ and unit-test product edits. This branch is limited
to tests/e2e per contributor instruction
* fix(e2e): harden batch list and team member setup races
provider_fallback list falls back when managed batches reject provider
filtering. Team create waits for /team/info and member_add retries on
transient team-not-found so split control-plane lag does not red the suite
* fix(e2e): remove .env.example
Leave local .env and docker-compose env wiring as the secret source
* fix(e2e): wire files_settings and faster budget rescheduler for compose
OpenAI/Azure batch file uploads need files_settings; budget reset e2e needs a
short rescheduler window. Drop unsupported bedrock-encoded create_batch cells,
tolerate bedrock file.bytes=0, and surface team-info wait failures instead of
hanging silently
* chore(e2e): strip verbose comments from batch capabilities
* fix(e2e): assert managed list fallback before provider_fallback skip
When provider-scoped list is rejected, still fetch the unfiltered list and
check the envelope. Only skip membership when the id is a raw
provider_fallback batch that managed list cannot index
At threshold 0.8 azure gpt-realtime fires speech-stop and creates a response, but the committed audio is clipped enough that the response comes back empty (0 transcript, 0 audio), failing the audio-input assertions deterministically. Dropping to 0.5 captures the full utterance so the model produces real content. Verified against a live proxy: 0.8 yields empty responses, 0.5 yields transcript and audio. openai tolerated 0.8; azure did not
azure batch used azure/gpt-4.1-mini-batch; gpt-4.1-mini is deprecating (2026-11-04)
and can no longer be deployed, so point it at gpt-5.4-mini (Global Batch) and bump
the api_version to 2025-04-01-preview. Requires an Azure Global Batch deployment
named gpt-5.4-mini-batch plus AZURE_API_BASE/AZURE_API_KEY on the proxy.
xai/grok-4-1-fast-non-reasoning is deprecated (2026-05-15); update the commented
xai realtime provider and the coverage-matrix doc to xai/grok-4-1-fast.
* fix: rust ocr tests finally pass
* fix: move realtime dir
* fix(realtime): normalize azure realtime api_base to host for Foundry endpoints
The azure realtime handler appended the realtime path to api_base verbatim, so a
Foundry base carrying a project path (.../api/projects/<name>) produced an invalid
realtime URL and the websocket handshake hung. Normalize api_base to scheme and host
before building the realtime path so both Azure OpenAI and Foundry bases connect
Point the e2e realtime azure deployment at the GA gpt-realtime model and stop passing
the os.environ refs the realtime path never unwraps, resolving them from the gateway
env by name instead. Drop the local docker-compose scaffolding from the tree
* test(e2e): add Gateway.list_files and list_fine_tuning_jobs for the discovery suite
The discovery endpoints suite calls client.gateway.list_files and
list_fine_tuning_jobs, which did not exist on Gateway, so both tests errored with
AttributeError before reaching the proxy. Add the two GET wrappers using the
existing FileListResponse / FineTuningJobsResponse models
* revert(realtime): drop azure realtime api_base host-normalization
The azure realtime handshake failure was a config issue, not a litellm bug: the
realtime base was set to the Azure AI Foundry project endpoint (.../api/projects/<p>),
but the OpenAI-compatible realtime route lives at the resource root. litellm correctly
appends the realtime path to whatever base it is given, so pointing the realtime
deployment at the resource root is the fix and no core change is needed
* fix(ocr): route azure_ai doc-intelligence to its own endpoint at the source
get_llm_provider inherits AZURE_AI_API_BASE into api_base for every azure_ai/* OCR
model, but Azure Document Intelligence is a separate resource reached via
AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, so doc-intelligence requests went to the wrong
host. Stop inheriting the azure_ai base for doc-intelligence models so api_base stays
unset and both the rust bridge and the python get_complete_url fall back to the
document-intelligence endpoint. This drops the earlier _rust_bridge_api_base reorder,
which only covered the rust path and let the env silently override an explicit api_base
* refactor(ocr): consolidate azure doc-intelligence detection; keep explicit api_base
Extract is_azure_document_intelligence_model as the single source of truth for the azure_ai doc-intelligence sub-route so the check is no longer duplicated across _prepare_ocr_request and _rust_bridge_api_base, and gate the dynamic_api_base suppression on the caller not supplying an api_base so an explicit endpoint is always honoured. Restore xai to the realtime PROVIDERS as a documented disabled entry instead of dropping it silently, and add a regression test pinning doc-intelligence api_base resolution.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
create_model now waits until the new deployment is servable on the data plane
(polls /v1/models) before returning, instead of assuming /model/new makes it
instantly callable. On a split control/data-plane proxy the gateway only sees a
model after its next DB reload, so an immediate call raced the reload and 400'd
with "Invalid model name passed" (embeddings, responses, messages, ocr, ...).
It also stops pinning model_info.id to the model_name, letting the proxy assign a
unique model_id. Re-registering a fixed-name deployment (the batch suite's
openai-batch et al.) after a failed teardown no longer collides on the model_id
unique constraint (prisma UniqueViolationError surfaced as the generic 500
"Failed to add model to db", erroring every batch_lifecycle case at setup)
* fix(ui): scope key models dropdown options to the key's team
A teamless key no longer offers the all-team-models option in the create and
edit forms; the backend expands that sentinel to the full proxy model list when
no team is attached, which is rarely what the user intended. A team key no
longer surfaces the all-proxy-models sentinel that leaks in verbatim when the
team's own model list carries it; the dropdown keeps All Team Models plus the
team's individual models.
Adds browser coverage to the management e2e suite: playwright (an optional
dependency behind importorskip) drives the proxy-served dashboard at /ui,
asserts the dropdown options a real user sees for teamless and team keys on
both create and edit, and walks the create modal end to end, reading the
persisted key back through /key/info.
* fix(ui): offer all-proxy-models on teamless keys in the models dropdown
A teamless key has no team allowlist to inherit, so the dropdown now offers All
Proxy Models in place of All Team Models on both the create and edit forms, with
the same exclusive-selection handling. Component and browser e2e tests updated to
pin the swapped option pair; the teamless create case now also walks the modal end
to end and reads the persisted key back through /key/info.
* test(ui): update no-team key creation spec to pick All Proxy Models
The create modal no longer offers All Team Models without a team; the teamless
path now offers All Proxy Models, which is what this spec exercises
* fix(ui): gate All Team Models on the team object being loaded
When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): filter all-proxy-models from teamless model fetch in key edit form
The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Introduce the e2e coverage denominator: 282 behavior cells across the six
tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance,
Logging & Guardrails, Other), one validated YAML row each, plus a collector
that diffs the registry against @pytest.mark.covers markers and reports
coverage per module.
The registry rows validate against a pydantic discriminated union so a row
cannot carry a field from another module. The collector is static: a
collect-only pass reads the markers, so it runs no test and needs no live
proxy. Register the covers marker suite-wide so that pass works under
--strict-markers.
This is a draft for review. Tiers are proposed rather than signed off, and a
few cells still need a support check or a prune.
* test(e2e): add management suite covering key/team/user/org lifecycle and route permissions
* test(e2e): decouple the enforcement-flip assertion from upstream health
Polling for a 200 on the newly-allowed model required it to be a routable,
healthy upstream, which is not the contract under test; poll until the
key_model_access_denied 403 lifts instead, excluding 401 so a revoked key
cannot read as success. Also document that the delete test's deferred teardown
firing on an already-deleted key is deliberate: cleanup must survive the test
failing before the in-body delete, and the repeat delete is a warn-free no-op
(the proxy answers 404 No keys found)
* test(e2e): inline the management suite's model and tpm literals
* test(e2e): drop the models_mgmt suite line from the folder list
* test(e2e): write the tpm limit as a plain integer literal
* fix(e2e): route model management to the control plane and restore Gateway.create_model
The split-transport routing table listed only /model/info as a control-plane
prefix, so /model/new and /model/delete were sent to the data-plane gateway,
which does not serve management routes and 404s them. Every suite that
registers deployments at runtime (llm_translation, batches, access_control)
failed on the split stage deployment because of this. Widen the prefix to
/model/ so all model-management routes reach the control plane while /models
stays on the data plane.
Separately, batch_client.py and several llm_translation tests call
gateway.create_model, but Gateway never had that method, so all 17 batch tests
errored at fixture setup with AttributeError. Add create_model/delete_model to
Gateway (with the optional mode that batches needs) and make EndpointsClient
delegate to it instead of carrying its own copy.
Regression tests cover both: the routing predicate for management vs LLM paths
and the Gateway model-management surface via a typed fake Transport. Both fail
on the previous code
* test(e2e): make the fake transport payload depend on response_type
The recording fake always answered with {"model_id": ...} even when the
caller asked for NoBody, which only validated because pydantic ignores extra
fields by default. Return an empty payload for response types that carry no
fields so a future extra="forbid" on NoBody cannot turn the delete test into
a ValidationError inside the fake
* test(e2e): probe the full spend read surface including schema-hidden routes
The curated spend-route list missed twelve read endpoints, most of them
include_in_schema=False and therefore invisible to the schema-discovery test:
/spend/logs/v2, /spend/logs/session/ui, /global/all_end_users,
/global/activity/exceptions/deployment, and the per-entity daily activity
family (user, user aggregated, team, organization, customer, end_user, tag).
Add them all, verified responsive against the live split stage deployment.
/end_user was missing from CONTROL_PLANE_PREFIXES, so /end_user/daily/activity
would have been routed to the data plane and 404ed like /model/new used to;
add the prefix and pin it plus the daily-activity routes in the transport
routing test.
/provider/budgets stays excluded with a documented reason: it returns 500
whenever router_settings.provider_budget_config is absent, so probing it on a
proxy without provider budget routing configured can never be green
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects
spend_tracking/spend_e2e_client.py imported SpendTagsResponse and
TagSpend from models, but neither was ever defined, so importing the
client raised ImportError and pytest aborted collection for the whole
e2e session. The tag-spend tests had never run.
Model /spend/tags as it actually answers: a bare array of per-tag
aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the
existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field
that also wouldn't match the array shape; it now reads .root, matching
how spend_logs consumes its RootModel.
* test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction
Adds regression nets and gap-surfacing tests:
A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the
DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document
that reasoning_effort='none' and thinking type='disabled' are silently dropped
(LIT-3686 / GH #27453)
A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py):
parametrized regression net asserting real completion content, not just a 200,
across the configured providers for /chat/completions and /responses (GH #28991)
A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is
honored and prompt-cache read tokens grow on a repeated cacheable prefix
A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call
rate limiter runs, then asserts no unattributed spend row is left behind by the
internal input-file retrieval (LIT-3266)
A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct
key_alias and asserts each alias gets its own labeled series on /metrics
A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict)
regression proving eviction must not close an httpx client still held by an
in-flight caller (LIT-3221 / GH #13034)
Extends tests/e2e/models.py with the typed request and response fields these
tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage
fields, spend-log api_key)
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): drop unused litellm-regression-tests submodule
The e2e suite migrated the regression cases into this repo; nothing
imports the submodule at runtime (only a provenance comment references
it), so the .gitmodules entry and gitlink pointing at a personal repo
would just make upstream CI init a submodule it never uses. Remove both
to keep the change test-only.
* test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage
The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client
close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread
cleanup vs shared client teardown) and belongs in its own PR, not this e2e
coverage PR, so revert the file to its base state.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): migrate access-control and inference-endpoint regression tests
Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first
access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior
llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config
* Update endpoints_client.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* chore(e2e): untrack gateway config and document e2e test location
Stop tracking tests/e2e/gateway/litellm-config.yml so the local proxy config stays on the machine
Add a note to CLAUDE.md that new e2e tests belong in tests/e2e/ and must follow that directory's conventions
* chore(e2e): add self-contained docker compose stack for local runs
Ship a docker-compose.yml that starts the proxy with a throwaway Postgres and Redis and inlines the proxy config with example models, so contributors can bring up a local gateway with nothing but a .env. Update CONTRIBUTING.md to match the inline-config flow
* chore(e2e): drop the second gemini deployment; one key is enough locally
* docs(e2e): make pre-commit steps ordered and require flagging internally found issues
* test(e2e): add vertex_ai passthrough spend-log coverage
Port the de-flake of the SDK-based vertex spend test (#31689) into the
tests/e2e/llm_translation harness. The vertexai SDK intermittently ignored the
proxy api_endpoint override and billed Vertex directly, so the request never
reached LiteLLM and no spend was logged; driving native generateContent over the
shared transport always reaches the proxy, which the harness already guarantees.
The vertex deployment is added at runtime through /model/new with
use_in_pass_through rather than declared in the gateway config, and deleted on
teardown. That registers the deployment's service account for the /vertex_ai
route, so the passthrough call sends only its litellm virtual key in
x-litellm-api-key and no upstream bearer, and the proxy mints the Vertex token
itself. The credential is the one the proxy already holds, read from the same
VERTEXAI_CREDENTIALS/VERTEXAI_PROJECT env; the test never mints a token.
Asserts both that the forward succeeds and that a costed SpendLogs row lands
(vertex_ai provider, a gemini model, spend > 0, call_type pass_through_endpoint),
correlated by the x-litellm-call-id header.
* Update tests/e2e/llm_translation/test_vertex_passthrough_e2e.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/e2e/llm_translation/test_vertex_passthrough_e2e.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
tests/e2e/CLAUDE.md captures the harness code-style rules (suite-as-a-class, shared transport, typed pydantic models, Result/unwrap, markers, typing) and the coverage-registry naming grammar; CONTRIBUTING.md gets the Contributors Guide intro and a Setup section
