mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
11 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
Sameer Kankute
|
38709ba9bb
|
feat(proxy): skip disable_background_health_check models on GET /health when flag set (#27716)
* feat(proxy): skip disable_background_health_check models on GET /health when flag set Co-authored-by: Cursor <cursoragent@cursor.com> * fix comment * fix greptile comments * Fix health check fallback kwargs * Format health endpoint * Harden direct health check kwargs compatibility for monkeypatched perform_health_check Replace substring-based TypeError detection with unexpected-keyword checks and a short retry chain (full kwargs, instrumentation only, filter only, minimal) so partial stubs work regardless of which optional kwarg fails first. Add proxy unit tests for legacy three-arg stubs and single-kwarg variants. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix black --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
harish876
|
d20c70f24c |
Optimize database query which fetches latest model_id, model_name pairs and dedupes them in memory.
Current fix includes - Updates test case - Optimized query with docstring. The change leverages deduplication and sorting logic from SQL - Added a bench script to differentiate peak memory usage before and after |
||
|
ishaan-berri
|
51876292a0
|
Litellm ishaan april4 2 (#25150)
* feat(router): integrate allowed_fails_policy into health check failures (#24988) * feat(router): integrate allowed_fails_policy into health check failures Health check failures now increment the same per-deployment failure counters used by allowed_fails_policy, so users can control how many health check failures of each error type are required before a deployment enters cooldown. - ahealth_check() preserves the original exception in its return dict - run_with_timeout() returns a litellm.Timeout on health check timeout - _perform_health_check() propagates exceptions to unhealthy endpoints - _write_health_state_to_router_cache() calls _set_cooldown_deployments for each unhealthy endpoint that has an exception - When allowed_fails_policy is set, the binary health check filter is bypassed so cooldown is the sole routing exclusion mechanism - Safety net: if all deployments are in cooldown with enable_health_check_routing=True, the cooldown filter is bypassed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(router): add health_check_ignore_transient_errors flag When enabled, health check failures with 429 (rate limit) or 408 (timeout) status codes are skipped from the cooldown pipeline. These are transient load issues, not broken deployments. Auth errors (401), 404, and 5xx errors still increment counters and trigger cooldown as before. Config (general_settings): health_check_ignore_transient_errors: true Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(router): also exclude 429/408 from health state cache when ignore_transient_errors set The previous fix only skipped cooldown counter increments. The health state cache was still marking 429/408 endpoints as is_healthy=False, causing the binary health check filter to exclude them from routing. Now, when health_check_ignore_transient_errors=True, 429/408 endpoints are also excluded from the unhealthy list passed to build_deployment_health_states(), so the binary filter treats them as unaffected (not unhealthy). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(router): add health check driven routing guide New standalone page covering the full health check routing feature: allowed_fails_policy integration, health_check_ignore_transient_errors, architecture SVG, step-by-step setup, and gotchas (TTL, AllowedFails semantics). Replaces the inline section in health.md with a link to the new page. Added to the Routing & Load Balancing sidebar. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): fix three CI failures - Add "exception" to ILLEGAL_DISPLAY_PARAMS in health_check.py so the exception object is stripped before the health endpoint serializes results to JSON (fixes TypeError: 'URL' object is not iterable) - Add allowed_fails_policy = None to FakeRouter stubs in test_router_health_check_routing.py (fixes AttributeError) - Add health_check_ignore_transient_errors to config_settings.md router settings reference table (fixes documentation test) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix litellm/tests/proxy_unit_tests/test_proxy_server.py * fix(router): address greptile review comments - Narrow cooldown safety-net bypass: only fires when allowed_fails_policy is set (cooldown is health-check driven). Without a policy, cooldowns are from real request failures and must not be bypassed. - Restore cooldown deployments DEBUG log that was accidentally removed. - Fix test_health TypeError: move exception extraction to a separate exceptions_by_model_id dict returned alongside endpoints, so exception objects never appear in the endpoint dicts that get JSON-serialized by the /health response. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): properly isolate exceptions from health response Return exceptions_by_model_id as a separate third value from _perform_health_check / perform_health_check so exception objects (which contain non-JSON-serializable httpx URL types) never appear in the endpoint dicts that get serialized by the /health response. Callers updated: _health_endpoints.py, shared_health_check_manager.py, proxy_server.py background loop. All use the exceptions dict only for cooldown integration, not for display. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(shared-health-check): fix remaining 2-value return sites and update type annotation * fix(health-check-routing): fix P0 cooldown integration never firing The cooldown loop was reading endpoint.get("exception") which is always None because exceptions are now returned via exceptions_by_model_id, not stored in endpoint dicts. Fixed to use _exceptions.get(model_id). Also fixes the transient-error filter to use _exceptions instead of endpoint.get("exception"), and fixes all remaining 2-value return sites in shared_health_check_manager.py. Tests updated to pass exceptions via exceptions_by_model_id parameter instead of endpoint dicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): fix P1 transient-error filter broken on cache hits When SharedHealthCheckManager returns cached results, exceptions_by_model_id is always {} so the transient-error filter defaulted to status 500 for all endpoints, incorrectly marking 429/408 endpoints as unhealthy. Fix: store integer exception_status on each unhealthy endpoint dict in _perform_health_check. _get_endpoint_exception_status() uses the live exception object when available (direct path) and falls back to the stored integer (cache-hit path). The integer is JSON-serializable and survives the shared cache round-trip. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): gate cooldown loop behind allowed_fails_policy Without the policy, cooldown is not the routing exclusion mechanism. Firing _set_cooldown_deployments for all enable_health_check_routing users was a backwards-incompatible change — 401s would immediately cooldown deployments that the binary filter would have recovered on the next cycle. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * revert: undo allowed_fails_policy gate on cooldown loop Cooldown integration via health checks is intentional for all enable_health_check_routing users, not just those with allowed_fails_policy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(docs+tests): fix health_check_ignore_transient_errors doc section and test coverage - Move health_check_ignore_transient_errors from router_settings to general_settings in config_settings.md (code reads it from general_settings) - Remove duplicate enable_health_check_routing / health_check_staleness_threshold entries that were incorrectly listed under router_settings - Replace TestHealthCheckEndpointExceptionPropagation tests with ones that exercise the real _perform_health_check code path via mocked ahealth_check, verifying exceptions appear in exceptions_by_model_id and NOT in endpoint dicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tests+docs): fix tuple unpacking and docs test failures - Update test mocks that return (healthy, unhealthy) to return (healthy, unhealthy, {}) to match the new 3-value signature - Update test unpackings of perform_shared_health_check to use healthy, unhealthy, _ = ... - Add health_check_ignore_transient_errors to router_settings section in config_settings.md (it is a Router constructor param, so the doc test requires it there; it also lives in general_settings for proxy use) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix CodeQL errors * fix(tests): fix 2-value unpackings of _perform_health_check in test_health_check.py * fix(tests): fix mock _perform_health_check returning 2-tuple instead of 3 * fix team routing --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add distributed lock for key rotation job (#23364) * fix: add distributed lock for key rotation job * fix: address Greptile review feedback on key rotation lock (#23834) * fix: address Greptile review feedback on key rotation lock * fix req changes greptile * feat(proxy): Optional on_error for guardrail pipeline (API / technical failures) (#24831) * guardrails fallback * docs * docs: add LITELLM_KEY_ROTATION_LOCK_TTL_SECONDS to environment variables reference * fix(mypy): accept Union[Dict, Any] in _get_deployment_order and use typed list to fix min() type error * fix(mypy): use Optional[str] for api_base in PydanticAI provider to match superclass signature --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: Shivam Rawat <shivam@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
Sameer Kankute
|
0debe92605 | Fix_mapped tests part 2 | ||
|
Alejandro Tapia
|
82f6d0fe43 | healthcheck-model_id-fix: There was quite a bit of code that needed to be changed since health checks were entirely keyed by model name. This includes, proxy logic, the dashboard, and even networking, because model name was the identifier everywhere. All changed files had tests added to them, which are passing with no regressions. | ||
|
Sameer Kankute
|
3d6b7f0d3d | Add background health checks to db | ||
|
Cole McIntosh
|
7d1e0651d8
|
Implement health check backend API and storage functionality - fix ci/cd (#11852)
* feat: Add health check functionality and endpoints - Introduced methods for saving health check results to the database, including validation and cleaning of data. - Added new health check endpoints to retrieve health check history and latest health statuses for models. - Updated model prices and context window configuration for new Azure transcription models. * test: Add unit tests for health check functionality - Introduced tests for PrismaClient health check methods, including saving results and retrieving health check history. - Added tests for the _save_health_check_to_db function to ensure proper handling of healthy and unhealthy endpoints. - Implemented mock objects to simulate database interactions and validate method behaviors. * Refactor health endpoint model ID handling and improve logging - Updated health endpoint to use `get_deployment` for retrieving model names based on model IDs, enhancing error handling for missing models. - Changed health check result saving to the database to be non-blocking by using `asyncio.create_task`. - Cleaned up code for better readability and maintainability. * Refactor utility functions in proxy module for improved readability and error handling - Removed unused imports and simplified exception handling in `_get_redoc_url` and `_get_docs_url` functions to manage circular imports. - Cleaned up logging statements for consistency and clarity. - Streamlined error message formatting in `handle_exception_on_proxy` function. * Enhance type hinting and default values in ProxyUpdateSpend class for improved clarity and robustness - Added type hints for `_end_user_list_transactions` to specify it as a dictionary mapping end user IDs to spend amounts. - Updated default values for optional fields in `SpendLogsPayload` to ensure they are initialized properly, enhancing error handling. - Refactored `_premium_user_check` function to improve model validation logic and error handling. * Fix disable_spend_updates method to handle None return value gracefully - Updated the disable_spend_updates method to return False if the environment variable DISABLE_SPEND_UPDATES is not set or is None, improving robustness in configuration handling. * Refactor join_paths function in utils.py for improved path handling - Enhanced the join_paths function to better manage leading and trailing slashes, ensuring correct path concatenation. - Added logic to handle cases where either base_path or route is empty, improving robustness and usability. * Enhance health check functionality and improve error handling - Introduced a new method `_save_health_check_to_db` for saving health check results to the database, utilizing safe JSON functions for data integrity. - Refactored existing health check methods to streamline the process and improve error logging. - Updated email sending logic to ensure secure connections and better error handling. - Improved spend update logic with batch processing and retry mechanisms for database operations. - Added utility functions for projected spend calculations and enhanced validation for team configurations. * Add health check methods for database interaction - Introduced `save_health_check_result` method to save health check results with detailed logging and validation. - Added `get_health_check_history` method for retrieving health check records with optional filtering. - Implemented `get_all_latest_health_checks` method to fetch the latest health checks for each model. - Enhanced error handling and logging for all new methods to improve reliability and traceability. * Refactor health check result saving to use typed arguments - Updated the `_save_health_check_to_db` function to call `save_health_check_result` with explicitly typed arguments instead of a dictionary spread, enhancing code clarity and type safety. - Removed unused method bindings in the mock Prisma client tests to streamline the test setup. * Remove unused `_save_health_check_to_db` function from utils.py to streamline code and improve maintainability. * Implement response time validation and details cleaning in health check result saving - Added `_validate_response_time` method to ensure response time values are valid and handle exceptions gracefully. - Introduced `_clean_details` method to validate and clean details JSON, improving data integrity. - Refactored `save_health_check_result` to utilize these new methods for optional fields, enhancing code clarity and maintainability. - Updated tests to bind new methods to the mock Prisma client for comprehensive testing. * Add health check utility functions and refactor existing endpoints - Introduced `_convert_health_check_to_dict` to standardize health check record conversion to dictionary format for JSON responses. - Added `_check_prisma_client` helper function to streamline database availability checks and improve error handling. - Refactored health check endpoints to utilize the new utility functions, enhancing code clarity and maintainability. * Refactor health check tests for improved clarity and coverage - Simplified the mock PrismaClient setup by consolidating method bindings. - Updated health check result saving tests to use parameterized scenarios for better coverage. - Added tests for health check history retrieval and graceful handling when no database client is provided. - Removed redundant mock functions to streamline the test suite. * Implement helper function for health check and database saving - Added `_perform_health_check_and_save` to encapsulate health check execution and optional database saving. - Refactored health endpoint logic to utilize the new helper function, improving code clarity and reducing redundancy. - Enhanced error handling and streamlined the process of saving health check results to the database. * refactor: rename target_model parameter to model in health check function |
||
|
Ishaan Jaff
|
1c1e41c51d |
Revert "Implement health check backend API and storage functionality (#11678)"
This reverts commit
|
||
|
Cole McIntosh
|
5f34ceea1a
|
Implement health check backend API and storage functionality (#11678)
* feat: Add health check functionality and endpoints - Introduced methods for saving health check results to the database, including validation and cleaning of data. - Added new health check endpoints to retrieve health check history and latest health statuses for models. - Updated model prices and context window configuration for new Azure transcription models. * test: Add unit tests for health check functionality - Introduced tests for PrismaClient health check methods, including saving results and retrieving health check history. - Added tests for the _save_health_check_to_db function to ensure proper handling of healthy and unhealthy endpoints. - Implemented mock objects to simulate database interactions and validate method behaviors. * Refactor health endpoint model ID handling and improve logging - Updated health endpoint to use `get_deployment` for retrieving model names based on model IDs, enhancing error handling for missing models. - Changed health check result saving to the database to be non-blocking by using `asyncio.create_task`. - Cleaned up code for better readability and maintainability. * Refactor utility functions in proxy module for improved readability and error handling - Removed unused imports and simplified exception handling in `_get_redoc_url` and `_get_docs_url` functions to manage circular imports. - Cleaned up logging statements for consistency and clarity. - Streamlined error message formatting in `handle_exception_on_proxy` function. * Enhance type hinting and default values in ProxyUpdateSpend class for improved clarity and robustness - Added type hints for `_end_user_list_transactions` to specify it as a dictionary mapping end user IDs to spend amounts. - Updated default values for optional fields in `SpendLogsPayload` to ensure they are initialized properly, enhancing error handling. - Refactored `_premium_user_check` function to improve model validation logic and error handling. * Fix disable_spend_updates method to handle None return value gracefully - Updated the disable_spend_updates method to return False if the environment variable DISABLE_SPEND_UPDATES is not set or is None, improving robustness in configuration handling. * Refactor join_paths function in utils.py for improved path handling - Enhanced the join_paths function to better manage leading and trailing slashes, ensuring correct path concatenation. - Added logic to handle cases where either base_path or route is empty, improving robustness and usability. * Enhance health check functionality and improve error handling - Introduced a new method `_save_health_check_to_db` for saving health check results to the database, utilizing safe JSON functions for data integrity. - Refactored existing health check methods to streamline the process and improve error logging. - Updated email sending logic to ensure secure connections and better error handling. - Improved spend update logic with batch processing and retry mechanisms for database operations. - Added utility functions for projected spend calculations and enhanced validation for team configurations. * Add health check methods for database interaction - Introduced `save_health_check_result` method to save health check results with detailed logging and validation. - Added `get_health_check_history` method for retrieving health check records with optional filtering. - Implemented `get_all_latest_health_checks` method to fetch the latest health checks for each model. - Enhanced error handling and logging for all new methods to improve reliability and traceability. * Refactor health check result saving to use typed arguments - Updated the `_save_health_check_to_db` function to call `save_health_check_result` with explicitly typed arguments instead of a dictionary spread, enhancing code clarity and type safety. - Removed unused method bindings in the mock Prisma client tests to streamline the test setup. * Remove unused `_save_health_check_to_db` function from utils.py to streamline code and improve maintainability. * Implement response time validation and details cleaning in health check result saving - Added `_validate_response_time` method to ensure response time values are valid and handle exceptions gracefully. - Introduced `_clean_details` method to validate and clean details JSON, improving data integrity. - Refactored `save_health_check_result` to utilize these new methods for optional fields, enhancing code clarity and maintainability. - Updated tests to bind new methods to the mock Prisma client for comprehensive testing. * Add health check utility functions and refactor existing endpoints - Introduced `_convert_health_check_to_dict` to standardize health check record conversion to dictionary format for JSON responses. - Added `_check_prisma_client` helper function to streamline database availability checks and improve error handling. - Refactored health check endpoints to utilize the new utility functions, enhancing code clarity and maintainability. * Refactor health check tests for improved clarity and coverage - Simplified the mock PrismaClient setup by consolidating method bindings. - Updated health check result saving tests to use parameterized scenarios for better coverage. - Added tests for health check history retrieval and graceful handling when no database client is provided. - Removed redundant mock functions to streamline the test suite. * Implement helper function for health check and database saving - Added `_perform_health_check_and_save` to encapsulate health check execution and optional database saving. - Refactored health endpoint logic to utilize the new helper function, improving code clarity and reducing redundancy. - Enhanced error handling and streamlined the process of saving health check results to the database. |