- Updated MidStreamFallbackError to retrieve and maintain the original status code from the wrapped exception.
- Ensured that message, request, and response fields remain consistent after calling the parent constructor.
- Added unit tests to verify the correct propagation of status codes and attributes in various scenarios.
Extract and preserve provider-specific headers (llm_provider-*) when
handling error responses from LLM providers. This ensures that useful
debugging information from providers is available even when requests
fail with BadRequestError or similar exceptions.
- Add `should_wrap_with_default_message` parameter to GuardrailRaisedException
- Update Generic Guardrail API to use clean error messages without wrapper
- When should_wrap_with_default_message=False, exception shows the original
blocked_reason directly (e.g., "pii detected") instead of verbose format
- Update test to verify GuardrailRaisedException is raised with clean message
* fix(router.py): add acompletion_streaming_iterator inside router
allows router to catch errors mid-stream for fallbacks
Work for https://github.com/BerriAI/litellm/issues/6532
* fix(router.py): working mid-stream fallbacks
* fix(router.py): more iterations
* fix(router.py): working mid-stream fallbacks with fallbacks set on router
* fix(router.py): pass prior content back in new request as assistant prefix message
* fix(router.py): add a system prompt to help guide non-prefix supporting models to use the continued text correctly
* fix(common_utils.py): support converting `prefix: true` for non-prefix supporting models
* fix: reduce LOC in function
* test(test_router.py): add unit tests for new function
* test: add basic unit test
* fix(router.py): ensure return type of fallback stream is compatible with CustomStreamWrapper
prevent client code from breaking
* fix: cleanup
* test: update test
* fix: fix linting error
* fix(exception_mapping_utils.py): correctly pass through 504 status code
openai also raises a 504 status code
* build(model_prices_and_context_window.json): add gpt-4o-mini-tts to model cost map
Fixes https://github.com/BerriAI/litellm/issues/9591
* fix(cost_calculator.py): fix input cost calculation for gpt-4o-mini-tts
Fixes https://github.com/BerriAI/litellm/issues/9591
* test: testing updates
* fix(main.py): fix key leak error when unknown provider given
don't return passed in args if unknown route on embedding
* fix(main.py): remove instances of {args} being passed in exception
prevent potential key leaks
* test(code_coverage/prevent_key_leaks_in_codebase.py): ban usage of {args} in codebase
* fix: fix linting errors
* fix: remove unused variable
* feat(main.py): mock_response() - support 'litellm.ContextWindowExceededError' in mock response
enabled quicker router/fallback/proxy debug on context window errors
* feat(exception_mapping_utils.py): extract special litellm errors from error str if calling `litellm_proxy/` as provider
Closes https://github.com/BerriAI/litellm/issues/7259
* fix(user_api_key_auth.py): specify 'Received Proxy Server Request' is span kind server
Closes https://github.com/BerriAI/litellm/issues/7298
* add the logprobs param for fireworks ai (#6915)
* add the logprobs param for fireworks ai
* (feat) pass through llm endpoints - add `PATCH` support (vertex context caching requires for update ops) (#6924)
* add PATCH for pass through endpoints
* test_pass_through_routes_support_all_methods
* sonnet supports pdf, haiku does not (#6928)
* (feat) DataDog Logger - Add Failure logging + use Standard Logging payload (#6929)
* add async_log_failure_event for dd
* use standard logging payload for DD logging
* use standard logging payload for DD
* fix use SLP status
* allow opting into _create_v0_logging_payload
* add unit tests for DD logging payload
* fix dd logging tests
* (feat) log proxy auth errors on datadog (#6931)
* add new dd type for auth errors
* add async_log_proxy_authentication_errors
* fix comment
* use async_log_proxy_authentication_errors
* test_datadog_post_call_failure_hook
* test_async_log_proxy_authentication_errors
* (feat) Allow using include to include external YAML files in a config.yaml (#6922)
* add helper to process inlcudes directive on yaml
* add doc on config management
* unit tests for `include` on config.yaml
* bump: version 1.52.16 → 1.53.
* (feat) dd logger - set tags according to the values set by those env vars (#6933)
* dd logger, inherit from .envs
* test_datadog_payload_environment_variables
* fix _get_datadog_service
* build(ui/): update ui build
* bump: version 1.53.0 → 1.53.1
* Revert "(feat) Allow using include to include external YAML files in a config.yaml (#6922)"
This reverts commit 68e59824a3.
* LiteLLM Minor Fixes & Improvements (11/26/2024) (#6913)
* docs(config_settings.md): document all router_settings
* ci(config.yml): add router_settings doc test to ci/cd
* test: debug test on ci/cd
* test: debug ci/cd test
* test: fix test
* fix(team_endpoints.py): skip invalid team object. don't fail `/team/list` call
Causes downstream errors if ui just fails to load team list
* test(base_llm_unit_tests.py): add 'response_format={"type": "text"}' test to base_llm_unit_tests
adds complete coverage for all 'response_format' values to ci/cd
* feat(router.py): support wildcard routes in `get_router_model_info()`
Addresses https://github.com/BerriAI/litellm/issues/6914
* build(model_prices_and_context_window.json): add tpm/rpm limits for all gemini models
Allows for ratelimit tracking for gemini models even with wildcard routing enabled
Addresses https://github.com/BerriAI/litellm/issues/6914
* feat(router.py): add tpm/rpm tracking on success/failure to global_router
Addresses https://github.com/BerriAI/litellm/issues/6914
* feat(router.py): support wildcard routes on router.get_model_group_usage()
* fix(router.py): fix linting error
* fix(router.py): implement get_remaining_tokens_and_requests
Addresses https://github.com/BerriAI/litellm/issues/6914
* fix(router.py): fix linting errors
* test: fix test
* test: fix tests
* docs(config_settings.md): add missing dd env vars to docs
* fix(router.py): check if hidden params is dict
* LiteLLM Minor Fixes & Improvements (11/27/2024) (#6943)
* fix(http_parsing_utils.py): remove `ast.literal_eval()` from http utils
Security fix - https://huntr.com/bounties/96a32812-213c-4819-ba4e-36143d35e95b?token=bf414bbd77f8b346556e
64ab2dd9301ea44339910877ea50401c76f977e36cdd78272f5fb4ca852a88a7e832828aae1192df98680544ee24aa98f3cf6980d8
bab641a66b7ccbc02c0e7d4ddba2db4dbe7318889dc0098d8db2d639f345f574159814627bb084563bad472e2f990f825bff0878a9
e281e72c88b4bc5884d637d186c0d67c9987c57c3f0caf395aff07b89ad2b7220d1dd7d1b427fd2260b5f01090efce5250f8b56ea2
c0ec19916c24b23825d85ce119911275944c840a1340d69e23ca6a462da610
* fix(converse/transformation.py): support bedrock apac cross region inference
Fixes https://github.com/BerriAI/litellm/issues/6905
* fix(user_api_key_auth.py): add auth check for websocket endpoint
Fixes https://github.com/BerriAI/litellm/issues/6926
* fix(user_api_key_auth.py): use `model` from query param
* fix: fix linting error
* test: run flaky tests first
* docs: update the docs (#6923)
* (bug fix) /key/update was not storing `budget_duration` in the DB (#6941)
* fix - store budget_duration for keys
* test_generate_and_update_key
* test_update_user_unit_test
* fix user update
* (fix) handle json decode errors for DD exception logging (#6934)
* fix JSONDecodeError
* handle async_log_proxy_authentication_errors
* fix test_async_log_proxy_authentication_errors_get_request
* Revert "Revert "(feat) Allow using include to include external YAML files in a config.yaml (#6922)""
This reverts commit 5d13302e6b.
* (docs + fix) Add docs on Moderations endpoint, Text Completion (#6947)
* fix _pass_through_moderation_endpoint_factory
* fix route_llm_request
* doc moderations api
* docs on /moderations
* add e2e tests for moderations api
* docs moderations api
* test_pass_through_moderation_endpoint_factory
* docs text completion
* (feat) add enforcement for unique key aliases on /key/update and /key/generate (#6944)
* add enforcement for unique key aliases
* fix _enforce_unique_key_alias
* fix _enforce_unique_key_alias
* fix _enforce_unique_key_alias
* test_enforce_unique_key_alias
* (fix) tag merging / aggregation logic (#6932)
* use 1 helper to merge tags + ensure unique ness
* test_add_litellm_data_to_request_duplicate_tags
* fix _merge_tags
* fix proxy utils test
* fix doc string
* (feat) Allow disabling ErrorLogs written to the DB (#6940)
* fix - allow disabling logging error logs
* docs on disabling error logs
* doc string for _PROXY_failure_handler
* test_disable_error_logs
* rename file
* fix rename file
* increase test coverage for test_enable_error_logs
* fix(key_management_endpoints.py): support 'tags' param on `/key/update` (#6945)
* LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965)
* fix(factory.py): ensure tool call converts image url
Fixes https://github.com/BerriAI/litellm/issues/6953
* fix(transformation.py): support mp4 + pdf url's for vertex ai
Fixes https://github.com/BerriAI/litellm/issues/6936
* fix(http_handler.py): mask gemini api key in error logs
Fixes https://github.com/BerriAI/litellm/issues/6963
* docs(prometheus.md): update prometheus FAQs
* feat(auth_checks.py): ensure specific model access > wildcard model access
if wildcard model is in access group, but specific model is not - deny access
* fix(auth_checks.py): handle auth checks for team based model access groups
handles scenario where model access group used for wildcard models
* fix(internal_user_endpoints.py): support adding guardrails on `/user/update`
Fixes https://github.com/BerriAI/litellm/issues/6942
* fix(key_management_endpoints.py): fix prepare_metadata_fields helper
* fix: fix tests
* build(requirements.txt): bump openai dep version
fixes proxies argument
* test: fix tests
* fix(http_handler.py): fix error message masking
* fix(bedrock_guardrails.py): pass in prepped data
* test: fix test
* test: fix nvidia nim test
* fix(http_handler.py): return original response headers
* fix: revert maskedhttpstatuserror
* test: update tests
* test: cleanup test
* fix(key_management_endpoints.py): fix metadata field update logic
* fix(key_management_endpoints.py): maintain initial order of guardrails in key update
* fix(key_management_endpoints.py): handle prepare metadata
* fix: fix linting errors
* fix: fix linting errors
* fix: fix linting errors
* fix: fix key management errors
* fix(key_management_endpoints.py): update metadata
* test: update test
* refactor: add more debug statements
* test: skip flaky test
* test: fix test
* fix: fix test
* fix: fix update metadata logic
* fix: fix test
* ci(config.yml): change db url for e2e ui testing
* bump: version 1.53.1 → 1.53.2
* Updated config.yml
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: paul-gauthier <69695708+paul-gauthier@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Sara Han <127759186+sdiazlor@users.noreply.github.com>
* fix(exceptions.py): ensure ratelimit error code == 429, type == "throttling_error"
Fixes https://github.com/BerriAI/litellm/pull/6973
* fix(utils.py): add jina ai dimensions embedding param support
Fixes https://github.com/BerriAI/litellm/issues/6591
* fix(exception_mapping_utils.py): add bedrock 'prompt is too long' exception to context window exceeded error exception mapping
Fixes https://github.com/BerriAI/litellm/issues/6629
Closes https://github.com/BerriAI/litellm/pull/6975
* fix(litellm_logging.py): strip trailing slash for api base
Closes https://github.com/BerriAI/litellm/pull/6859
* test: skip timeout issue
---------
Co-authored-by: ershang-dou <erlie.shang@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: paul-gauthier <69695708+paul-gauthier@users.noreply.github.com>
Co-authored-by: Sara Han <127759186+sdiazlor@users.noreply.github.com>
* docs(exception_mapping.md): add missing exception types
Fixes https://github.com/Aider-AI/aider/issues/2120#issuecomment-2438971183
* fix(main.py): register custom model pricing with specific key
Ensure custom model pricing is registered to the specific model+provider key combination
* test: make testing more robust for custom pricing
* fix(redis_cache.py): instrument otel logging for sync redis calls
ensures complete coverage for all redis cache calls
