mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
ec8088f064
|
test(e2e): vendor API testing coverage (#34557)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly |
||
|
Yuneng Jiang
|
472dd2716f
|
revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)"
This reverts commit
|
||
|
mubashir1osmani
|
dcb4e5033c
|
test(e2e): vendor API strategy coverage across endpoints (#34649)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly * fix(e2e): address greptile and veria review on vendor strategy suite Store search tool keys as os.environ refs and resolve them in SearchAPIRouter. Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells * fix(e2e): drop search_api_router os.environ expansion from vendor suite Keep the PR test-only. Search tools register without an api_key so the proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a. * test(e2e): drop search e2e suite from vendor strategy PR Remove the /v1/search coverage file and its registry rows so this PR no longer carries search endpoint testing. |
||
|
mubashir1osmani
|
ac5b51253a
|
test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149)
* test(e2e): add other suite covering master-key auth and health lifecycle Covers the other.* holding-pen cells that were uncovered: master-key valid_allows/invalid_denied on the admin /user/list gate, and the lifecycle probes liveness.ping, readiness.public_probe, readiness.reports_db_status, and readiness_details.authenticated_diagnostics. New tests/e2e/other/ suite on the shared ProxyClient; the health probes send no auth header to prove the public routes need no credential, and the details route is asserted to reject an anonymous caller while exposing version/db diagnostics to the master key. * test(e2e): cover block_code_execution and openai_moderation guardrails Extends the guardrails suite with two built-in guardrails registered per request (default_on=False, opted in via the chat body's guardrails selector) so neither intercepts unrelated traffic on the shared proxy. block_code_execution.pre_call.blocks: a python code block plus a run-this request is intercepted with the canned content-blocked message and the model never runs, while the same code block asked about with don't-run-it reaches the model. Verified live. openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming the moderation policy while a benign prompt passes. The guardrail calls OpenAI's moderation API; verifying it needs an OpenAI key with moderation quota (this account currently 429s the moderation endpoint). Adds a shared create_backend_model helper and a generic register() plus per-request guardrails/max_tokens on the client so more built-ins can reuse the same path. * test(e2e): cover presidio PII masking (pre_call + post_call) Registers a presidio guardrail per request (default_on=False) with the analyzer/anonymizer bases supplied in the registration params, so the test controls its own dependency and needs no proxy restart. presidio.pre_call.masks: a repeat-verbatim request comes back with the <EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was anonymized before the model saw it. presidio.post_call.masks: with apply_to_output the model's own emitted email is masked on the way out, so the caller never receives the raw value. Both verified live against real presidio analyzer + anonymizer containers. logging_only is intentionally not covered: /spend/logs exposes no prompt messages to read back the masked log, and a logging_only run also masked the response, contradicting its contract; noted in the module docstring for a follow-up. * test(e2e): cover presidio logging_only masking via OTEL read-back Adds the third presidio cell, guardrail.presidio.logging_only.masks. The logging_only contract (mask what is logged, do not block) is verified by reading the request's gen-AI span back from the real OTEL destination: the span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder, never the raw email, and the call itself is not blocked. Reads the trace via the shared OtelReader, promoted from logging/ to the suite root so both suites use it. The masked prompt is polled to a deadline because logging_only masks the payload asynchronously and the span can briefly export before the mask lands. Drops the throwaway chat_send in favor of the existing transport.send for the call-id capture. * fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test Stage runs multiple gateway pods behind the shared key. POST /guardrails registers a new default-on guardrail in-process immediately only on the pod that served the create call; every other pod picks it up on its next periodic DB sync (proxy_server.py, every 30s), so the very next chat call can race a pod that has not synced yet. Poll to a 40s deadline instead of asserting on the first response, matching the existing pattern in test_budget_reset_advances_e2e.py. * test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call) Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed to the pre_mcp_call hook, and lets a clean argument reach the upstream server. The guardrail attaches with default_on because per-key/request guardrail selection is dropped from the synthetic MCP request the hook sees; the banned keyword is unique per run so default_on only intercepts this test's own call. mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call because the event type is rewritten for call_mcp_tool. Drives the tool directly via /mcp-rest/tools/call for a deterministic check of the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model invokes an MCP tool. * fix(e2e): mid-conversation messages test uses client.proxy not client.gateway EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the mid-conversation system test still referenced .gateway, which fails the e2e basedpyright gate. Aligns it with the rest of the harness. * test(e2e): address review on the guardrail coverage MCP tool-call guardrail: poll the banned call until the guardrail is enforced instead of asserting on the first call, so the control-plane -> data-plane guardrail sync cannot race the check into a false pass-through; add a repeat banned call after enforcement to guard against a partial-propagation state. OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no moderation quota) from a guardrail failure, so an account-capability gap reads as such rather than as "did not block". Runs green with a moderation-capable key. * test(e2e): close partial-propagation false-pass in MCP guardrail block test The single post-block repeat call could be load-balanced back to the same already-synced data-plane pod, so the test could pass while another pod still lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to the guardrail create time (every pod is guaranteed to have DB-synced only after a full ~30s sync interval), then require the banned call to stay blocked across several attempts; a pass-through after that window is a real leak, not a race. * test(e2e): drop xfail-style rate-limit branch from openai_moderation test OpenAI's /v1/moderations is free and returns 200 with the env key (verified directly), so the RateLimitedError branch mislabeled the failure: a 429 there is insufficient_quota (no account billing), not throttling. The branch also only printed a softer message before failing anyway, an xfail-in-disguise the e2e rules forbid. A 429 now falls through and fails loudly with the full result. |
||
|
mubashir1osmani
|
28f012bb52
|
test(true_rabbit): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping (#33843)
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly. * test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL, TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path, Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm chat, and Nova Sonic realtime. Registry cells updated for the new markers. * test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and Langfuse driver models now use Anthropic haiku so local runs stay green when Gemini daily quota is exhausted. * test(e2e): drop Langfuse spend suite; feature is being deprecated Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the langfuse-only conftest driver/credentials fixtures. * test(e2e): fold provider/batch feature tests into their endpoint suites Keep the e2e layout endpoint- and suite-scoped instead of one file per provider or feature Move the virtual-key auth case into access_control/test_access_control_e2e.py as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone test_virtual_key_auth_e2e.py Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py. The hosted_vllm batch case is skipped for now since it needs a live vLLM server (HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant Merge the cohere, gemini and hosted_vllm chat cases into llm_translation/test_chat_completions_regression_e2e.py so /chat/completions coverage lives in one endpoint file, and repoint the coverage_registry source fields to the new homes Move the shared CacheControl / TextBlock / RichMessage request blocks into the root models.py (re-exported from endpoints_client) so quota_management can use them without a cross-suite import, which also clears the basedpyright errors in test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed json.loads path * test(e2e): address review feedback and re-home virtual-key coverage Replace the tautological Bedrock assume-role batch id assertion (`startswith(...) or batch.id`, always true) with a managed-id shape check, since the unified target_model_names path re-encodes the id rather than returning a raw ARN Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no longer consume the key's sole request unit before batch create runs; the batch create then clears the generic per-request limiter and the batch limiter is what returns the "Batch rate limit exceeded" body the assertions check Set exercised_on to [] on the pass-through header test; it drives a pass-through endpoint, not /chat/completions Move the virtual-key valid_allows / invalid_denied cells from other.yaml to mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management, and point its covers marker at the new ids |
||
|
yuneng-jiang
|
a43f128a74
|
test(e2e): add coverage registry and collector (#32304)
Introduce the e2e coverage denominator: 282 behavior cells across the six tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance, Logging & Guardrails, Other), one validated YAML row each, plus a collector that diffs the registry against @pytest.mark.covers markers and reports coverage per module. The registry rows validate against a pydantic discriminated union so a row cannot carry a field from another module. The collector is static: a collect-only pass reads the markers, so it runs no test and needs no live proxy. Register the covers marker suite-wide so that pass works under --strict-markers. This is a draft for review. Tiers are proposed rather than signed off, and a few cells still need a support check or a prune. |