mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-09 22:31:41 +00:00
12 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
ryan-crabbe-berri
|
e9d40a8f73 |
test: enforce F811 so a duplicate definition cannot silently replace the first
A name bound twice keeps only the second binding. In `tests/` that is nearly always a repeated import, harmless but misleading, and the same rule is what catches the cases that are not harmless: a local that shadows an import the module still calls, and a second `def test_x` that quietly replaces the first. 311 of the 344 sites were repeated imports and came out with ruff's own fix. The remaining 33 needed a decision. Four modules imported a name they never used because a local definition below already shadowed it. Two comprehensions bound `call` over `unittest.mock.call`, which those modules import and use. One test rebound the two module handles its nested reload closure had captured. One class attribute shadowed an unused `status` import. The load-test fixtures move to a conftest, which is how pytest is meant to share them, so the test module no longer imports three fixture names it never calls. The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that fixture by name before the body runs, so the parameter never shadows anything. |
||
|
Deepanshu Lulla
|
0580465384
|
feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416)
* feat(router): add per-deployment allowed_fails_policy and cooldown_time override support Three bugs fixed in the router cooldown system: (1) deployment-level allowed_fails and allowed_fails_policy in model_info now take precedence over router-level settings in _should_cooldown_deployment; (2) failed fallback deployments now get evaluated for cooldown via _trigger_cooldown_for_failed_deployment, bypassing the Logging dedup gate; (3) DualCache promotes Redis cooldown entries using default 600s TTL instead of true remaining cooldown time -- _corrected_active_cooldown now evicts expired entries and corrects stale in-memory TTLs on backfill. Adds ServiceUnavailableError, BadGatewayError, and NotFoundError fields to AllowedFailsPolicy and cooldown_time to LiteLLMParamsTypedDict. * fix(router): gate fallback cooldown trigger on has_logged_async_failure; use only litellm_metadata for deployment ID * fix(router): use X | Y union syntax to fix UP007 strict lint gate * test(router_utils): add coverage for _trigger_cooldown_for_failed_deployment and has_logged_async_failure gate * test(router_utils): cover deployment cooldown override and exception swallow paths * fix(router): add InternalServerError/ServiceUnavailableError/BadGatewayError/NotFoundError to router-level get_allowed_fails_from_policy * fix(router): format router.py and add router-level policy tests * test(router): add CI-visible coverage for per-deployment cooldown policy Tests for `_get_deployment_cooldown_policy`, `_resolve_allowed_fails_from_policy`, and `_should_cooldown_based_on_deployment_policy` (cooldown_handlers.py), the `_corrected_active_cooldown` branches in CooldownCache, and the four new exception-type branches in `Router.get_allowed_fails_from_policy` (router.py) -- all in `tests/test_litellm/` which the enterprise-routing CI job runs. * fix(router): use is not None guard for cooldown_time_override in should_cooldown_based_on_allowed_fails_policy A cooldown_time_override of 0 was previously treated as falsy and silently fell through to the router-level cooldown_time value. Switched to an explicit is not None check so that zero is honored as a valid override. Added a regression test covering the zero case. * fix(router): honor has_logged_async_failure and metadata for fallback cooldown; support both model_info and litellm_params locations Manual verification against a live proxy surfaced that the fallback-cooldown-gap trigger never actually fired: the has_logged_async_failure check read a plain attribute that Logging never sets (the real flag lives in model_call_details), and the deployment_id lookup only trusted litellm_metadata, which regular chat completions never populate (only batch/thread/file endpoints do). Router overwrites model_info on whichever key is present before every attempt, so metadata is equally authoritative there, not caller-controlled as previously assumed. Also let allowed_fails/allowed_fails_policy/cooldown_time be set under either model_info or litellm_params, each preferring its own canonical location. * fix(router): fix ContentPolicyViolationError policy shadowing and partial-policy zero-threshold Two bugs from Greptile review on PR #34416: - ContentPolicyViolationError subclasses BadRequestError, so listing BadRequestError first in _EXCEPTION_POLICY_FIELDS made the isinstance check always match BadRequestError for content-policy errors, using the wrong allowed_fails threshold. Reordered so the subclass is checked first. - A deployment with a partial allowed_fails_policy and no deployment-wide allowed_fails forced allowed_fails_override=0 for any exception type its policy didn't cover, cooling the deployment down on the first unrelated failure. Now defers to router-level behavior for uncovered exception types instead of forcing an immediate cooldown. * fix(router): only trust a metadata/litellm_metadata bucket the router itself wrote deployment info into veria-ai flagged that preferring litellm_metadata whenever present could pick up a caller-supplied litellm_metadata.model_info.id (preserved via allow_client_pricing_override) instead of the metadata bucket the router actually populated for a regular completion's fallback attempt, naming an arbitrary "victim" deployment for cooldown. Router._update_kwargs_with_deployment() always writes model_info and deployment_model_name into the same bucket together. Only trust a bucket that carries deployment_model_name alongside model_info, since that marker is only ever set by the router itself, not by request-body metadata. * test(router): add regression coverage for ContentPolicyViolationError policy shadowing The subclass-ordering fix in commit 38fe4e4490 had no regression test. Verified the new test fails on the pre-fix ordering (asserts 2, got 10) before restoring the fix, and confirmed the same behavior through the full _should_cooldown_deployment call path against a real Router instance. * fix(router): let explicit allowed_fails_policy entries override the generic 4XX cooldown exclusion _is_cooldown_required skips cooldown evaluation for any 4XX status outside {429, 401, 408, 404} by default, since a generic client error is usually not the deployment's fault. BadRequestError and ContentPolicyViolationError both carry status 400, so their AllowedFailsPolicy fields (BadRequestErrorAllowedFails, ContentPolicyViolationErrorAllowedFails, both router-level pre-existing and the new deployment-level ones) were silently unreachable: an operator could set them to any value with no effect, since _is_cooldown_required blocked cooldown evaluation before that policy was ever consulted. _should_run_cooldown_logic now also checks whether an explicit allowed_fails_policy entry (deployment-level or router-level) covers the exception's type, and if so, proceeds with cooldown evaluation regardless of the generic status-code exclusion. The exclusion remains the default for exception types with no explicit policy. Verified live against a mock-triggered ContentPolicyViolationError (config-level mock_response, azure/gpt-4.1-mini deployment) with BadRequestErrorAllowedFails=100 and ContentPolicyViolationErrorAllowedFails=0 on the same deployment: it now cools down after exactly one ContentPolicyViolationError instead of never cooling down. * fix(router): use the router-stamped failed_deployment_id for fallback cooldown targeting Greptile flagged a real gap in the metadata-bucket-based deployment lookup: for a generic-API-call fallback, the router writes the current attempt into litellm_metadata, but a stale "metadata" bucket carrying the same deployment_model_name marker (from an earlier point) would be picked first, cooling the wrong deployment. Router already has a more robust, pre-existing mechanism for this exact problem: _set_failed_deployment_id_on_exception stamps the failing deployment's id directly onto the exception at the point of failure, immune to metadata-bucket ambiguity since a caller can't influence it and it doesn't depend on which bucket the current call type happens to use. It just wasn't called from _ageneric_api_call_with_fallbacks_helper's except block, unlike _completion/_acompletion. Added the missing call there (matching the existing pattern exactly), and changed _trigger_cooldown_for_failed_deployment to prefer exception.failed_deployment_id when present, falling back to metadata-bucket inspection only for call paths that don't stamp it yet. Verified live: the standard fallback-cooldown-gap scenario (two bad-key deployments in a fallback chain) still correctly cools down both the originally-called and fallback deployment. * fix(router): address human review on per-deployment cooldown overrides Scope allowed_fails_policy override to deployment-level only (a router-level policy predates this feature and must keep its existing behavior), exempt advisor-orchestration failures from the fallback cooldown trigger, keep the single-deployment model group protection intact against a generic deployment-level allowed_fails, make cooldown_time precedence consistent across resolution paths, fix a falsy-zero swallowing bug in the router-level allowed_fails fallback, and make allowed_fails_policy resolution fall through to the next matching exception type instead of stopping at the first unset field. Also restrict allowed_fails/allowed_fails_policy/cooldown_time to model_info: litellm_params gets copied into the actual provider request, so a router-only setting placed there would leak into that request. * test(router): update test_cooldown_handlers.py for the deployment-policy signature change Surfaced by the rebase: this mirrored test file (tests/test_litellm/ mirrors litellm/) predates the router_unit_tests/ coverage added earlier in this PR and was still calling _should_cooldown_based_on_deployment_policy with its old 4-argument signature and asserting the now-removed litellm_params cooldown_time location. * test(router): update test_fallback_event_handlers.py for model_info-only cooldown_time Another mirrored test file surfaced by the rebase that still asserted the now-removed litellm_params.cooldown_time location. * fix(router): match cooldown-duration precedence in the fallback path to the primary path _trigger_cooldown_for_failed_deployment only checked deployment config before falling back to the router default, skipping the response Retry-After header step that Router.deployment_callback_on_failure applies on the primary path. * fix(router): restore litellm_params.cooldown_time as a pre-existing fallback cooldown_time already had litellm_params support on Router.deployment_callback_on_failure before this PR; the earlier model_info-only restriction (aimed at the leak concern for the genuinely new allowed_fails/allowed_fails_policy fields) incorrectly dropped that pre-existing capability too. model_info still takes priority when both are set. * fix(router): keep the fallback-cooldown trigger in sync with #35104's review fixes Applies the same two fixes landed on the split-out PR #35104 (which #34416 still duplicates until it's rebased onto the merged base): increment the deployment's per-minute failure counter before evaluating cooldown, and require the server-stamped failed_deployment_id instead of trusting a metadata bucket, since neither "metadata" nor "litellm_metadata" can be told apart from a caller-supplied one without knowing the call's function_name. * fix(router): freeze the model_info fallback mapping to satisfy the type-discipline gate * fix(router): defer f-string interpolation in fallback-cooldown debug logs * fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget * fix(router): suppress reportPrivateUsage for cross-module cooldown helpers * fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks * fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one * fix(router): keep up with upstream typing modernization and Final-annotation ratchet * fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout * fix(router): stamp dynamic client-side-credential id in completion fallback paths too The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential) deployment id on exceptions, but the regular _completion/_acompletion exception handlers still stamped the static shared deployment's id. A tenant using invalid forwarded credentials could generate repeated failures attributed to, and eventually cooling down, the shared deployment other tenants rely on. Extracted the stamping logic into one shared helper used by all three call sites (generic API, sync completion, async completion) so the fix and future changes to it stay in one place. * fix(proxy): recognize body-supplied timeout/request_timeout/stream_timeout as caller-controlled client_side_timeout was only set when the caller used the x-litellm-timeout header, but Router._get_timeout also resolves the effective timeout from kwargs["timeout"], kwargs["request_timeout"], and kwargs["stream_timeout"], all settable directly in the request body (and x-litellm-stream-timeout wasn't marked either). A caller could set any of those to a near-zero value, force a 408 on every deployment in a fallback chain, and cool down deployments other tenants rely on without the guard in _trigger_cooldown_for_failed_deployment recognizing it as caller-controlled. Also strip any client-forged client_side_timeout from the request body so the marker is always server-computed. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
Mateo Wang
|
2c733c00f5
|
chore(ci): modernize model references in tests and configs (#27856)
* test: modernize models used in CircleCI e2e test suites
Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.
- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
(also aligning oai_misc_config model_name with what
test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
-> claude-sonnet-4-5-20250929
* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5
Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.
* test: modernize models across remaining CI-mounted configs & tests
Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).
Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
text-embedding-ada-002 underlying to text-embedding-3-small. User-
facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.
Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
+ paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
and test_bedrock_anthropic_messages_test.py: bump router fixtures
using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.
* test: modernize placeholder model literals in router_unit_tests
Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.
Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
claude-sonnet-4-5-20250929 / claude-opus-4-7 /
claude-haiku-4-5-20251001 as appropriate
Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers
* test: modernize placeholder model literals across remaining CI suites
Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.
Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
/ gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro
Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
translation/transformation logic). Only the deprecated 20250514
references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
equivalent).
- Top-level tests calling the proxy through user-facing aliases
(gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
in proxy_server_config.yaml stay; only the underlying model was
bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
is model-name handling).
- Fake / mock / openai/fake identifiers.
Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
(bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.
* test: fix CI failures from model modernization sweep
CI surfaced 4 categories of regression from the bulk modernization:
1. Azure deployment names are customer-specific. Reverted:
- tests/litellm_utils_tests/test_health_check.py: azure/text-
embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
account does not have a text-embedding-3-small deployment).
- tests/logging_callback_tests/test_custom_callback_router.py:
same revert for two router fixtures driving aembedding.
2. gpt-5 family does not accept temperature != 1. Tests that pass a
custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
non-reasoning OpenAI mini that still accepts temperature/logprobs):
- tests/logging_callback_tests/test_datadog.py
- tests/logging_callback_tests/test_langsmith_unit_test.py
- tests/logging_callback_tests/test_otel_logging.py
3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
tests/test_openai_endpoints.py::test_chat_completion_streaming
exercises logprobs/top_logprobs through that alias. Bumped the
underlying model to gpt-4.1 (non-reasoning, still modern).
4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
hardcoded model="gpt-4o" and a model-specific spend value. Reverted
the litellm.acompletion calls in the test to model="gpt-4o" so the
fixture's exact-match assertions still hold.
5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
anthropic.messages.create routing to openai/gpt-5-mini returned an
empty content[0] with max_tokens=100 (reasoning-token consumption).
Swapped to openai/gpt-4.1-mini.
* test: fix Assistants API model + 2 cursor[bot] review nits
1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
isn't accepted by the /v1/assistants endpoint
("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
API-supported, non-reasoning).
2. example_config_yaml/pass_through_config.yaml: the previous sweep
bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
Sonnet tier intact. (Cursor bugbot review.)
3. example_config_yaml/simple_config.yaml: model_name was left as
gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
muddles the "simple" example. Make both sides gpt-5-mini so the
most basic example is a straight 1:1 mapping again. (Cursor bugbot
review.)
* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models
tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
|
||
|
Devaj Mody
|
f25344484f
|
fix(router): add minimum request threshold for error rate cooldown (#17464)
Fixes #17418 - Add DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS constant (default: 5) - Require minimum requests before applying error rate cooldown - Prevents cooldown from triggering on first failure |
||
|
IQHL (Hans Jacob Landelius)
|
6633b33085 | formatting | ||
|
IQHL (Hans Jacob Landelius)
|
5b9d516e43 | unit test | ||
|
Jugal D. Bhatt
|
a46b9d376f
|
[Prometheus] Move Prometheus to enterprise folder (#12659)
* fix tools fetch for keys * add promethues to enterprise * remove old prom * remove old prom * fix tests * safe imports * add if * fix enterprise test * rename imports * added label import * added label import * move tests to enterprise * fix tests * add log * build: update versions --------- Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> |
||
|
Krish Dholakia
|
80f7af510b
|
Improve Proxy Resiliency: Cooldown single-deployment model groups if 100% calls failed in high traffic (#7823)
* refactor(_is_cooldown_required): move '_is_cooldown_required' into cooldown_handlers.py * refactor(cooldown_handlers.py): move cooldown constants into `.constants.py` * fix(cooldown_handlers.py): remove if single deployment don't cooldown logic move to traffic based cooldown logic Addresses https://github.com/BerriAI/litellm/issues/7822 * fix: add unit tests for '_should_cooldown_deployment' * test: ensure all tests pass * test: update test * fix(cooldown_handlers.py): don't cooldown single deployment models for anything besides traffic related errors * fix(cooldown_handlers.py): fix cooldown handler logic * fix(cooldown_handlers.py): fix check |
||
|
Ishaan Jaff
|
c7f14e936a
|
(code quality) run ruff rule to ban unused imports (#7313)
* remove unused imports * fix AmazonConverseConfig * fix test * fix import * ruff check fixes * test fixes * fix testing * fix imports |
||
|
Krish Dholakia
|
44e7ffd05c
|
(perf) Litellm redis router fix - ~100ms improvement (#6483)
* docs(exception_mapping.md): add missing exception types Fixes https://github.com/Aider-AI/aider/issues/2120#issuecomment-2438971183 * fix(main.py): register custom model pricing with specific key Ensure custom model pricing is registered to the specific model+provider key combination * test: make testing more robust for custom pricing * fix(redis_cache.py): instrument otel logging for sync redis calls ensures complete coverage for all redis cache calls * refactor: pass parent_otel_span for redis caching calls in router allows for more observability into what calls are causing latency issues * test: update tests with new params * refactor: ensure e2e otel tracing for router * refactor(router.py): add more otel tracing acrosss router catch all latency issues for router requests * fix: fix linting error * fix(router.py): fix linting error * fix: fix test * test: fix tests * fix(dual_cache.py): pass ttl to redis cache * fix: fix param * perf(cooldown_cache.py): improve cooldown cache, to store cache results in memory for 5s, prevents redis call from being made on each request reduces 100ms latency per call with caching enabled on router * fix: fix test * fix(cooldown_cache.py): handle if a result is None * fix(cooldown_cache.py): add debug statements * refactor(dual_cache.py): move to using an in-memory check for batch get cache, to prevent redis from being hit for every call * fix(cooldown_cache.py): fix linting erropr |
||
|
Ishaan Jaff
|
891e9001b5
|
(testing) add router unit testing for send_llm_exception_alert , router_cooldown_event_callback , cooldown utils (#6258)
* add router unit testing for send_llm_exception_alert * test router_cooldown_event_callback * test test_router_cooldown_event_callback_no_prometheus * test_router_cooldown_event_callback_no_deployment * test_router_cooldown_event_callback_no_deployment * add testing for test_should_run_cooldown_logic * test_increment_deployment_successes_for_current_minute_does_not_write_to_redis * test test_should_cooldown_deployment_allowed_fails_set_on_router * use helper for _is_allowed_fails_set_on_router * add complete testing for cooldown utils * move router unit tests * move router handle error * fix test_send_llm_exception_alert_no_logger |