mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
e07a7129c5
* feat(proxy): redact or drop individual batch records instead of rejecting the file A single record tripping a guardrail rejected the whole upload, which is unusable for a file holding thousands of rows. A record a guardrail rewrites is now submitted in its rewritten form, a record it blocks is left out, and the create response reports every changed record by both custom_id and line so a caller can reconcile against the file it sent. The same outcome is written to the proxy log and to request metadata, so it is not visible only to the caller. A rewritten record goes straight to a spool and only its offset is carried, so a masking guardrail touching most rows of a large upload does not build a second copy of the file on the heap, and the rewrite runs off the event loop the way the sibling full-file validation does. Both proxy-injected metadata keys are captured from the record and restored exactly, including an explicit null, so a masked row keeps the tags that decide how it is attributed. A record is dropped only when a guardrail judged its content. `GuardrailRaisedException` now carries `blocked_content` for that, because half its raise sites in the repo signal an unreachable or unparseable backend under a fail-closed policy, and treating those as blocks would turn "refuse this request" into "drop this record and submit the rest". The default is off, so a raise that does not say what it means aborts the upload instead of silently shrinking the file. * fix(proxy): only drop a batch record on a verdict the guardrail actually reached A guardrail that reports a technical failure as an HTTPException carrying a block status was read as a content block, so an unreachable backend under a fail-closed policy quietly shrank the file instead of failing the upload. Two in-tree integrations do exactly that, and one of them defaults to fail-closed, so the broken configuration was the default one. Such an exception is raised `from` the underlying error, which is a deliberate statement that something else caused it, and no content verdict in the repo is raised that way, so the chain now settles it. Implicit context is left alone, since a block raised inside an unrelated `except` would read as a failure. Two annotation errors in the same family: the one GuardrailRaisedException subclass in tree never opted into blocked_content, so a real block took the whole upload down with it, and straiker's block helper is reached both from its verdict and from its fail-closed handler, so it claimed a verdict for an outage. The helper now takes the flag from its caller. A record could also opt itself out of the chain. Guardrail selection reads a body-level `guardrails` key ahead of the proxy-injected list, and online that key can only add to the key and team selection, never replace it, so a batch record naming an empty list skipped every guardrail that was not default_on and was still reported as scanned. Every injected key is now stripped before dispatch and restored afterwards. A guardrail that reroutes a record to another model is honoured on the online path by rewriting the model, which the scan read as a rewrite and submitted in the same file, sending content to the provider the reroute existed to avoid. Every record of a batch file goes to one provider, so the upload is refused instead, naming the line. The scan spool is closed on the paths that never read it back. * fix(proxy): give the scan the metadata bag guardrails actually read, and close its spools The narrowed request metadata was installed under `litellm_metadata` only, but a record is scanned as the chat request it describes, and the guardrails that pick a policy from a request header read `metadata` instead. Noma choosing an application and Aim choosing a user both look there, so the header allowlist added for them did not reach either one and a batch record was still evaluated under the fallback policy. The scan metadata now goes into both bags, which are both stripped and restored, so neither survives into the record that ships. The scan spool was closed on the paths that abort, which are exactly the paths where it is empty, and left open on the one path where it holds the rewritten records. Nothing closed the rewrite output either, where before this feature the uploaded handle belonged to Starlette. The upload now owns both and closes them however it exits. * fix(proxy): register the scan spool before the rewrite can fail The scan spool was added to the request's cleanup list only after the rewrite returned, so a rewrite that raised, which for a spilled file can be as ordinary as the disk filling up, jumped to the handler with the list still empty and left the scan's own handle open. The rewrite also left its half-written output behind on that path, since nothing owns that handle until it is returned. Both now close. |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.