mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
d515a285b1
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger posted the whole queue as one body and cleared it in a finally block, so an oversize batch, a transient 5xx, or a failed token call dropped every queued record, and records logged while a send was in flight were cleared with it. Both the standard and the audit queue share the sender. Move Datadog's proactive size split and 413 halving into a shared helper, litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it with a 1MB size check. A lone record that still 413s is dropped, everything a transient failure leaves undelivered goes back to the front of its queue, and the retry queue is capped at max_queue_size so an unreachable workspace cannot grow memory without bound * fix(azure_sentinel): retry undelivered records on the flush timer only Requeued records made every later event cross the batch_size threshold, so a down ingestion endpoint got one full-queue resend per request. Threshold sends now go through flush_queue, so they take the flush lock instead of racing the timer, and they stand down while records are awaiting retry. A record that cannot be serialized raised out of the size probe and killed the periodic flush task. The probe now runs inside the failure handling, so the batch is split and only the record that cannot be serialized is dropped. * fix(azure_sentinel): decide threshold sends under the flush lock Concurrent callbacks all read logs_awaiting_retry before the first send finished, so each one resent the whole queue once that send failed. The flag and the batch_size threshold are now rechecked while holding the flush lock, and each queue sends only itself instead of going through flush_queue, which was retrying the other queue too. * test(azure_sentinel): cover successful threshold waiters * fix(azure_sentinel): preserve cancelled batches for retry * fix(azure_sentinel): requeue only the undelivered part of a cancelled split A batch over the ingestion cap goes out in pieces, so a cancellation partway through requeued pieces the destination had already accepted and sent them a second time on the next flush The split helper now raises a cancellation carrying the records it never delivered, and Azure Sentinel requeues those instead of the whole batch * fix(azure_sentinel): drop batches a permanent rejection will never accept A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request will fail the same way on every retry, so requeueing it held the batch, and every record logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429) still keep the whole batch, and a shared classifier gives Datadog the same rule The serialization probe now catches any exception, not just TypeError and ValueError, because safe_dumps hands pydantic models to model_dump and can raise anything. It also splits on record count, so a recovery flush sends batch_size records per request instead of serializing the whole requeued queue to measure it Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python 3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the BatchSendCancelled subclass escaped the logging worker as an unhandled error The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so a deployment with the cap at zero is not left waiting for a timer flush with nothing queued to retry * chore(logging): document mutable queue ownership Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean * fix(datadog): preserve non-413 retry behavior Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter * fix(batch_utils): requeue by default and let Sentinel opt into dropping The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections Also drops an explicit return None the strict ruff gate flags in the test helper |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| rust-python-harness | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_rust_python_harness.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.