mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
11 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
ec8088f064
|
test(e2e): vendor API testing coverage (#34557)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly |
||
|
Yuneng Jiang
|
09a98f5505
|
test(e2e): settle control-plane writes across every replica, not just one
The suite already waits for a new model or agent to become servable before handing it back, but that wait returns on the first successful read. Every request opens a fresh connection (e2e_http calls requests.* with no Session), so a load-balanced Service routes each one independently: one successful read proves one replica converged, and the caller's next request re-rolls and can land on a replica that has not reloaded yet. At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1 replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no healthy deployments for this model", and a /model/info listing that contained one of two models created moments apart. Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds (30s by default, 7s on the e2e stack) plus margin, and settle after every control-plane create whose object the suite then uses: - ProxyClient.create_model and A2AClient.register_agent, after their existing polls -- the poll still fails loudly if the object never appears at all - GuardrailsClient.register, which had no barrier; create_content_filter_guardrail and create_bedrock_guardrail now route through it instead of POSTing directly - the guardrail creates in mcp_client and logging_client - the vertex passthrough model, whose body cannot go through create_model Left alone: the /model/new calls that assert a 403 or read back a status code, since they never use the model. |
||
|
Yuneng Jiang
|
472dd2716f
|
revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)"
This reverts commit
|
||
|
mubashir1osmani
|
dcb4e5033c
|
test(e2e): vendor API strategy coverage across endpoints (#34649)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly * fix(e2e): address greptile and veria review on vendor strategy suite Store search tool keys as os.environ refs and resolve them in SearchAPIRouter. Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells * fix(e2e): drop search_api_router os.environ expansion from vendor suite Keep the PR test-only. Search tools register without an api_key so the proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a. * test(e2e): drop search e2e suite from vendor strategy PR Remove the /v1/search coverage file and its registry rows so this PR no longer carries search endpoint testing. |
||
|
mubashir1osmani
|
5182dfa66b
|
test(e2e): remove the Presidio guardrail suite (#35129)
Drops tests/e2e/guardrails/test_presidio_guardrail_e2e.py and the PresidioParamsBody it was the only caller of. Both cases were red on most stage runs between 07-25 and 07-29: pre_call failed 6 of 11 runs, post_call 6 of 11, with post_call reporting the raw address reaching the caller while apply_to_output was set. The cause was propagation, not masking. GuardrailsClient.register() posts /guardrails and returns immediately with no readiness wait, unlike ProxyClient._await_model_servable or GuardrailsClient._await_team, and the data plane only picks a new guardrail up on its next periodic DB sync. Calls issued before that sync pass the raw value through. #34833 has since made both cases poll to the deadline, and on the current build each masks on the first attempt, so the suite is expected to be green now; it is being removed because it spends real provider money on every retry and because a pod replaced mid-poll still reproduces the old failure. The three guardrail.presidio.* rows stay in coverage_registry/guardrail.yaml and go uncovered on purpose, so Presidio reads as a tier-P0 gap in Grafana rather than dropping out of the denominator. |
||
|
mubashir1osmani
|
a10365e84d
|
test(e2e): stop racing control-plane writes across the mcp, a2a, guardrail and passthrough suites (#34833)
* test(e2e): wait for MCP tool discovery instead of racing it
/v1/mcp/server returns as soon as the DB row is written, but the gateway runs
the initialize + tools/list handshake against the upstream lazily, on the first
request that needs it. Every MCP test read tools/list immediately after
registering, so it raced that handshake.
The gateway reports a server it has not discovered yet exactly like a dead one:
it catches the per-server handshake exception and returns an empty tool list.
The tests asserted on a single read, so the race surfaced as "granted key never
saw search_datadog_logs; tools=frozenset()" while a sibling test against the
same upstream in the same run passed.
Add McpClient.await_tool, which polls tools/list to the suite's existing
poll_timeout and returns the qualified tool name, and route the four discovery
sites through it. An unreachable upstream or an unapplied grant still fails, and
the failure now names the last tools/list result.
Refs LIT-4821
* test(e2e): wait for a2a agents to reach the data plane after registration
POST /v1/agents is a control-plane write; the /a2a/{agent_id} routes that serve
the card and run message/send are data plane and only see the agent after the
next DB reload. Every test registered an agent and immediately read its card or
sent it a message, so the first data-plane touch could 404 on the agent it had
just created.
register_agent now waits for the card to become servable before returning, the
same way ProxyClient.create_model waits for a new model, so callers do not each
have to poll. Registration failures skip the wait, leaving the two rejection
tests unchanged. A genuine propagation failure now fails naming the agent id and
the last card read rather than as a bare 404 on whichever /a2a call ran first.
Refs LIT-4821
* test(e2e): wait for presidio guardrails to sync before asserting masking
Registering a guardrail is a control-plane write; the data-plane worker that
serves /chat/completions only picks it up on its next periodic DB sync (~30s), so
the first call after the create ran against a worker with no guardrail and passed
the raw email straight through. The tests asserted on that first call, so they
read in-flight propagation as a PII leak.
Confirmed directly against a live proxy: the same call is unmasked at t=0s and
masked at t=8s, and the presidio analyzer itself correctly returns EMAIL_ADDRESS
with score 1.0 the whole time. The MCP guardrail suite already documents and
waits out this exact sync delay; presidio never got the same treatment.
Poll the call until the placeholder replaces the PII, so the assertions judge the
synced state. A guardrail that never masks still fails, on the last unmasked
content. pre_call and post_call now pass repeatably.
Refs LIT-4821
* test(e2e): drop the presidio logging_only check pending LIT-4841
pre_call and post_call masking both pass once the guardrail-sync wait is in place,
but logging_only left the raw email in the OTEL span's gen_ai.input.messages on
every attempt across a full poll deadline. Keeping an assertion against
known-failing behavior just turns every run red, so the cell is tracked in
LIT-4841 instead.
The registry row stays, so guardrail.presidio.logging_only.masks now reports as an
uncovered gap rather than silently disappearing.
Refs LIT-4821, LIT-4841
* test(e2e): wait for guardrail sync in bedrock, moderation and block-code checks
All three asserted on the first call after registering a guardrail, so they were
served by a data-plane worker that had not synced it yet (~30s DB poll) and read
in-flight propagation as a guardrail that failed to block. Verified directly: the
openai_moderation guardrail lets a flagged prompt through at t=0s and returns
"Violated OpenAI moderation policy" at t=8s.
The reasoning-only responses noted in triage (content=None with reasoning_tokens
set) were a symptom of the same thing, not the cause; these are pre_call
guardrails, so a synced guardrail rejects the request before the model runs.
Add poll_until_blocked to guardrails_client for the two that surface a non-success
status, and poll on the block marker in the block_code_execution check, which
replaces the reply rather than erroring. All eight guardrail tests now pass.
Refs LIT-4821
* test(e2e): drop the openai prompt-cache check pending LIT-4841
Prompt caching never engages through the proxy: cached_tokens is 0 on every
repeat, while the identical payload sent straight to OpenAI reports 3615 cached
tokens on the second call. Pinning prompt_cache_key on the proxy request restores
caching (3328 tokens), so something varying per request is defeating OpenAI's
automatic prefix cache.
That is a product bug with a direct billing cost, tracked in LIT-4841. The
registry row stays, so llm.chat_completions.openai.prompt_cache_5m.nonstream.works
now reports as an uncovered gap instead of failing every run.
Refs LIT-4821, LIT-4841
* test(e2e): drop the responses metadata redis-ttl check
It failed on a Redis read timeout against the stage serverless cache
(berrie-litellm-stage-ieib2i.serverless.use1.cache.amazonaws.com:6379), a
reachability problem this suite has hit before rather than a proxy defect the
assertion can pin down.
The file held only this test. Its other cell,
llm.responses.openai.basic.nonstream.works, is still covered by
test_responses_e2e.py; other.config.responses.metadata_redis_ttl_bounded becomes
an uncovered registry row, taking headline coverage 314/431 -> 312/431.
Refs LIT-4821
* test(e2e): fix passthrough header propagation and openai body, drop the cost check
Three separate problems behind the two passthrough failures.
The header test 404'd because POST /config/pass_through_endpoint is a
control-plane write and the worker serving the route only registers it on its next
config reload; measured at ~18s on a live proxy. Wait for the route to stop 404ing
before calling it. The readiness probe reuses the master key and omits
anthropic-version so polling does not bill a completion per attempt.
The openai passthrough body sent max_tokens, which the gpt-5 family rejects
outright ("Unsupported parameter: 'max_tokens' is not supported with this model").
Confirmed against OpenAI directly: max_tokens 400s, max_completion_tokens 200s.
Passthrough forwards the body untouched by design, so the body was simply wrong.
test_openai_passthrough_nonstreaming_logs_cost still finds no SpendLogs row for
its call_id after the fix, so it is removed rather than left red; the gemini and
anthropic passthrough cost checks still cover that path.
Passthrough suite is 8/8 green.
Refs LIT-4821
|
||
|
mubashir1osmani
|
64fc19d61a
|
fix(e2e): stop tests from breaking the shared proxy for every suite after them (#34664)
* fix(e2e): stop the cache-settings test from persisting a degraded Redis config TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache settings and wrote them back, intending a no-op. Its capture modelled only type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and `redis_startup_nodes`. That is not recoverable on its own. `/cache/settings` persists what it receives into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and init_cache_settings_in_db re-applies it on a timer, so a restart does not clear it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext standalone node and every Redis call blocks to socket timeout. On the affected deployment that took out rate limiting entirely (the v3 limiter is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag, per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`, whose last statement syncs guardrails and never ran. 60 of 72 failures in one run traced back here. The settings blob is now round-tripped verbatim via a RootModel over an exhaustive value union, so a subset cannot be written. Two guards make a regression fail loudly at this test instead of silently downstream: - refuse to write when GET reports redis_type=cluster but omits redis_startup_nodes, which is the exact precondition for persisting a downgrade. GET resolves the stored row overlaid with REDIS_* env and never reads YAML, so a cluster configured only in YAML cannot round-trip here - compare /cache/ping before and after, so a write that breaks connectivity fails this test rather than every suite that follows The underlying product defect is filed as LIT-4816: GET cannot express the effective config, and a partial POST is allowed to downgrade transport. This change only stops the suite from triggering it; the Admin UI can still do so. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param Two failures that had nothing to do with the guardrail or route under test. create_bedrock_guardrail registered with default_on=True, which applies the guardrail to every request the proxy serves. The upstream ApplyGuardrail call was answering 403, and that came back to unrelated traffic as `403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough headers test alongside the bedrock one. The harness already supports the per-request `guardrails` selector, so the guardrail is now registered opted out of default_on and selected by the test that wants it. A broken upstream guardrail fails its own test instead of whatever else is running. Note this only contains the blast radius; the 403 itself still needs the bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its own until that is sorted. The OpenAI passthrough body sent `max_tokens`, which newer models reject with "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead." Passthrough forwards the body untranslated, so drop_params does not apply and the body has to satisfy OpenAI's contract directly. vllm_chat keeps max_tokens, which vLLM accepts. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): drop the pinned a2a api_key that broke every message/send #34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent. The a2a bridge forwards the agent's litellm_params straight into litellm.acompletion() without expanding "os.environ/" indirection, so that literal string was sent upstream as x-api-key and every message/send failed with `AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`. Omitting api_key restores the normal provider resolution: litellm reads ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what the agent-owner flow depends on and what the suite did before #34512. Verified against a live proxy, same agent shape each time: api_key omitted -> message/send 200 api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key api_key <literal key> -> message/send 200 and the key itself is valid (direct call to api.anthropic.com returns 200), so this was indirection that never got expanded rather than a bad credential. This accounts for four failures (test_semver_protocol_version_registers_and_serves, test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape, test_pinned_v1_0_serves_nested_message_shape). They were previously reported as `403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail short-circuited the request before it ever reached the bridge and hid this. The bridge silently ignoring "os.environ/" in agent params is a product defect in its own right, filed separately; anyone configuring an agent credential that way through the UI hits the same wall. basedpyright clean (0 errors) under the e2e gate. * test(e2e): make the load suite less aggressive against a shared proxy 750 users at spawn rate 50 saturated the request path hard enough to distort the latency-sensitive suites sharing the same proxy, and it spends real provider money at that rate. Drop to 200 users at spawn rate 20. The RPS floor moves with the user count rather than staying put, so the assertion keeps its meaning instead of becoming a formality: 355 RPS over 750 users is ~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a similar pass margin. A request-path regression still trips it. All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE, E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run. Note the recorded failure for this test was "no requests completed in 60s", which was the gateway wedged on unreachable Redis rather than a throughput regression; this change is about not perturbing its neighbours, not about that failure. * fix(e2e): make the reasoning-tokens assertion exercise a request that reasons test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then asserted reasoning_tokens > 0. The model answers that directly without reasoning, so 0 is correct behavior and the assertion was testing the model's discretion rather than litellm's reporting. Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how the test provisions its model: reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0 reasoning_effort=high, the prompt used here -> reasoning_tokens=114 Raised to high effort with a prompt that requires a proof plus a search, so the field under test is actually populated and the assertion fails only if litellm stops surfacing it. While confirming this I also checked prompt caching, which needed no change: cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix against a dedicated deployment. An earlier reading of 0 was an artifact of probing a fan-out alias whose requests land on different deployments, not a caching defect. * test(e2e): skip the files-list test while LIT-4820 is open GET /v1/files does not include a just-uploaded file. The upload returns 200 and GET /v1/files/{id} resolves it, but the listing never contains it: the returned set stays fixed at 27 entries whose newest created_at is roughly ten hours older than the upload, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window. Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the suite red and masking a new regression somewhere else in the same test. The assertion is left exactly as it was on purpose. It encodes the contract we actually want, that a file retrievable by id is also enumerable, and anything that lists files (a UI picker, cleanup tooling that lists then deletes and would therefore leak provider-side files) depends on it. Relaxing it to get green would delete the signal. The skip reason says so and links the ticket, and the ticket records that removing this marker is part of its definition of done. Matches the existing pattern in this file, where test_unified_file_and_batch_create skips with a reason citing LIT-3266. While skipped, the registry cell llm.files.openai.list.nonstream.works has no passing covering test, so files-list coverage reports as uncovered rather than passing, which is the honest state. * fix(e2e): parse Sentinel node lists in the cache-settings model The value union covered scalar lists and lists of mappings, but not lists of lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on a Sentinel deployment pydantic rejected the response: sentinel_nodes.list[dict[str,...]].1 Input should be a valid dictionary [input_value=['localhost', 26380]] The round-trip test reads GET /cache/settings before it writes anything, so that rejection failed the test at the read, before any assertion ran. A Sentinel deployment would have looked like a broken cache-settings route rather than a model too narrow to parse a documented shape. A list element may now be a scalar, a list or a mapping, which covers both node shapes without special-casing either and tolerates a heterogeneous list instead of rejecting the whole response. Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain node, url mode with a null discrete field) plus transport() key selection. Confirmed it fails on the previous union and passes on this one: old union -> 1 failed, 4 passed (the sentinel case) new union -> 5 passed * test(e2e): remove the cache-settings round-trip test The test could not fail for the thing it claimed to test, and could break the deployment it ran against. Both halves of that are worth stating. It read the live settings, wrote back identical values, and asserted the read-back matched. If POST /cache/settings were a complete no-op that returned 200 and touched nothing, GET would still return the values read a moment earlier and the test would pass. It verified that GET is stable, not that the route persists anything. Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig, that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a plaintext standalone client and every later Redis call blocks to socket timeout. On 2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing, Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved, and guardrail sync never ran. Guarding the previous shape was not sufficient. Writing the blob verbatim plus a cluster precondition and a /cache/ping check narrowed the hazard but did not remove it, because GET cannot express the effective config: it resolves the stored row overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row written that drops it. No round-trip through this route is safe on a shared proxy. Removed with the models and helpers it owned, and TestCacheSettingsModel with them since it existed only to protect that parsing. The registry row mgmt.cache_settings.update.happy_path stays, now carrying the rationale for why it is deliberately uncovered and what a safe test would require (an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade transport). Coverage therefore reports this cell as a gap, which is the honest state. Collector passes --strict; the module still collects 11 tests. |
||
|
mubashir1osmani
|
fa9e0f180c
|
test(e2e): make the bedrock guardrail test match the guardrail it points at (#34568)
The bedrock guardrail e2e test could never pass on stage. Two reasons. It sent a bomb-making prompt expecting "stock hate/violence filters" to block, but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no contentPolicy at all; it denies the topic and words "bread"/"cake" plus profanity. ApplyGuardrail returns action=NONE for the old prompt, so the request passes and the test reports "default-on guardrail did not block". Send a prompt the configured policy actually denies instead. It also registered the guardrail with aws_access_key_id / aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those env vars are deliberately absent from the gateway (static AWS keys hijack RDS IAM auth), and guardrail litellm_params do not expand os.environ/ indirection, so the literal string reached boto and failed with "Invalid AWS region format: 'os.environ/AWS_REGION'". Drop all three and let the gateway sign ApplyGuardrail with its own pod-identity role, which is how the standard stack is meant to reach Bedrock. Verified against the live stage proxy: registering the guardrail with only identifier/version and sending the new prompt returns 400 "Violated guardrail policy", satisfying both assertions. |
||
|
mubashir1osmani
|
b5bc3631e1
|
refactor(e2e): drop require_env, read os.environ where a cred is used (#34413)
require_env hard-failed a test (and, for the shared litellm-ops secret, drove piling every provider credential into one blob) whenever an optional cred was absent. Most call sites either read a value the test actually uses or just gated on the runner's env for a key the gateway consumes. Read os.environ directly where the test uses the value; drop the presence-only gates so those cases run against the proxy instead of pre-failing on the runner's environment. Removes the require_env helper from e2e_config. |
||
|
mubashir1osmani
|
ac5b51253a
|
test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149)
* test(e2e): add other suite covering master-key auth and health lifecycle Covers the other.* holding-pen cells that were uncovered: master-key valid_allows/invalid_denied on the admin /user/list gate, and the lifecycle probes liveness.ping, readiness.public_probe, readiness.reports_db_status, and readiness_details.authenticated_diagnostics. New tests/e2e/other/ suite on the shared ProxyClient; the health probes send no auth header to prove the public routes need no credential, and the details route is asserted to reject an anonymous caller while exposing version/db diagnostics to the master key. * test(e2e): cover block_code_execution and openai_moderation guardrails Extends the guardrails suite with two built-in guardrails registered per request (default_on=False, opted in via the chat body's guardrails selector) so neither intercepts unrelated traffic on the shared proxy. block_code_execution.pre_call.blocks: a python code block plus a run-this request is intercepted with the canned content-blocked message and the model never runs, while the same code block asked about with don't-run-it reaches the model. Verified live. openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming the moderation policy while a benign prompt passes. The guardrail calls OpenAI's moderation API; verifying it needs an OpenAI key with moderation quota (this account currently 429s the moderation endpoint). Adds a shared create_backend_model helper and a generic register() plus per-request guardrails/max_tokens on the client so more built-ins can reuse the same path. * test(e2e): cover presidio PII masking (pre_call + post_call) Registers a presidio guardrail per request (default_on=False) with the analyzer/anonymizer bases supplied in the registration params, so the test controls its own dependency and needs no proxy restart. presidio.pre_call.masks: a repeat-verbatim request comes back with the <EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was anonymized before the model saw it. presidio.post_call.masks: with apply_to_output the model's own emitted email is masked on the way out, so the caller never receives the raw value. Both verified live against real presidio analyzer + anonymizer containers. logging_only is intentionally not covered: /spend/logs exposes no prompt messages to read back the masked log, and a logging_only run also masked the response, contradicting its contract; noted in the module docstring for a follow-up. * test(e2e): cover presidio logging_only masking via OTEL read-back Adds the third presidio cell, guardrail.presidio.logging_only.masks. The logging_only contract (mask what is logged, do not block) is verified by reading the request's gen-AI span back from the real OTEL destination: the span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder, never the raw email, and the call itself is not blocked. Reads the trace via the shared OtelReader, promoted from logging/ to the suite root so both suites use it. The masked prompt is polled to a deadline because logging_only masks the payload asynchronously and the span can briefly export before the mask lands. Drops the throwaway chat_send in favor of the existing transport.send for the call-id capture. * fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test Stage runs multiple gateway pods behind the shared key. POST /guardrails registers a new default-on guardrail in-process immediately only on the pod that served the create call; every other pod picks it up on its next periodic DB sync (proxy_server.py, every 30s), so the very next chat call can race a pod that has not synced yet. Poll to a 40s deadline instead of asserting on the first response, matching the existing pattern in test_budget_reset_advances_e2e.py. * test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call) Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed to the pre_mcp_call hook, and lets a clean argument reach the upstream server. The guardrail attaches with default_on because per-key/request guardrail selection is dropped from the synthetic MCP request the hook sees; the banned keyword is unique per run so default_on only intercepts this test's own call. mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call because the event type is rewritten for call_mcp_tool. Drives the tool directly via /mcp-rest/tools/call for a deterministic check of the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model invokes an MCP tool. * fix(e2e): mid-conversation messages test uses client.proxy not client.gateway EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the mid-conversation system test still referenced .gateway, which fails the e2e basedpyright gate. Aligns it with the rest of the harness. * test(e2e): address review on the guardrail coverage MCP tool-call guardrail: poll the banned call until the guardrail is enforced instead of asserting on the first call, so the control-plane -> data-plane guardrail sync cannot race the check into a false pass-through; add a repeat banned call after enforcement to guard against a partial-propagation state. OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no moderation quota) from a guardrail failure, so an account-capability gap reads as such rather than as "did not block". Runs green with a moderation-capable key. * test(e2e): close partial-propagation false-pass in MCP guardrail block test The single post-block repeat call could be load-balanced back to the same already-synced data-plane pod, so the test could pass while another pod still lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to the guardrail create time (every pod is guaranteed to have DB-synced only after a full ~30s sync interval), then require the banned call to stay blocked across several attempts; a pass-through after that window is a real leak, not a race. * test(e2e): drop xfail-style rate-limit branch from openai_moderation test OpenAI's /v1/moderations is free and returns 200 with the env key (verified directly), so the RateLimitedError branch mislabeled the failure: a 429 there is insufficient_quota (no account billing), not throttling. The branch also only printed a softer message before failing anyway, an xfail-in-disguise the e2e rules forbid. A 429 now falls through and fails loudly with the full result. |
||
|
mubashir1osmani
|
28f012bb52
|
test(true_rabbit): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping (#33843)
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly. * test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL, TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path, Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm chat, and Nova Sonic realtime. Registry cells updated for the new markers. * test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and Langfuse driver models now use Anthropic haiku so local runs stay green when Gemini daily quota is exhausted. * test(e2e): drop Langfuse spend suite; feature is being deprecated Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the langfuse-only conftest driver/credentials fixtures. * test(e2e): fold provider/batch feature tests into their endpoint suites Keep the e2e layout endpoint- and suite-scoped instead of one file per provider or feature Move the virtual-key auth case into access_control/test_access_control_e2e.py as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone test_virtual_key_auth_e2e.py Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py. The hosted_vllm batch case is skipped for now since it needs a live vLLM server (HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant Merge the cohere, gemini and hosted_vllm chat cases into llm_translation/test_chat_completions_regression_e2e.py so /chat/completions coverage lives in one endpoint file, and repoint the coverage_registry source fields to the new homes Move the shared CacheControl / TextBlock / RichMessage request blocks into the root models.py (re-exported from endpoints_client) so quota_management can use them without a cross-suite import, which also clears the basedpyright errors in test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed json.loads path * test(e2e): address review feedback and re-home virtual-key coverage Replace the tautological Bedrock assume-role batch id assertion (`startswith(...) or batch.id`, always true) with a managed-id shape check, since the unified target_model_names path re-encodes the id rather than returning a raw ARN Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no longer consume the key's sole request unit before batch create runs; the batch create then clears the generic per-request limiter and the batch limiter is what returns the "Batch rate limit exceeded" body the assertions check Set exercised_on to [] on the pass-through header test; it drives a pass-through endpoint, not /chat/completions Move the virtual-key valid_allows / invalid_denied cells from other.yaml to mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management, and point its covers marker at the new ids |