Commit graph

5 commits

Author SHA1 Message Date
Mateo Wang
d0c82c308d
fix(main): stop per-request custom pricing from clobbering shared model_cost pricing (#32163)
* fix(main): stop per-request custom pricing from clobbering shared model_cost pricing

A request routed through a wildcard deployment with explicit zero pricing
(e.g. openai/* with input_cost_per_token: 0) registered that pricing on the
shared {provider}/{model} key in litellm.model_cost, so sibling deployments
relying on built-in pricing logged $0 until process restart (LIT-3991).

Request-time registration in completion()/embedding() now mirrors the
router-startup isolation: router-originated requests register full pricing
under the deployment's unique model id only, while the shared backend key
receives the entry with custom pricing fields stripped. Direct SDK calls
without a router deployment id keep the legacy shared-key registration.

The stripping logic is shared via
CustomPricingLiteLLMParams.strip_custom_pricing_fields and reused by
Router._create_deployment and Router.add_deployment.

* test: update legacy tests that asserted per-request pricing leaking into shared model_cost

test_router_fallbacks_with_custom_model_costs asserted the shared
claude-sonnet-4-5-20250929 entry ends up with the deployment's 30/60
pricing, which is exactly the cross-deployment leak this PR removes; it
now asserts the shared key keeps the built-in pricing, matching the
test's stated goal.

test_cost_calc.py::test_run computed streaming cost via
completion_cost(response), which only matched the non-stream cost while
the shared gpt-3.5-turbo entry was poisoned with the per-request
2/token pricing; it now passes the request's custom pricing explicitly
via custom_cost_per_token.
2026-07-07 10:25:31 -07:00
Krrish Dholakia
48d5acf678 test(test_cost_calc.py): fix test to handle llm api errors 2024-12-24 16:49:02 -08:00
Krish Dholakia
27a4d08604
Litellm dev 2024 12 19 p3 (#7322)
* fix(utils.py): remove unsupported optional params (if drop_params=True) before passing into map openai params

Fixes https://github.com/BerriAI/litellm/issues/7242

* test: new test for langfuse prompt management hook

Addresses https://github.com/BerriAI/litellm/issues/3893#issuecomment-2549080296

* feat(main.py): add 'get_chat_completion_prompt' customlogger hook

allows for langfuse prompt management

Addresses https://github.com/BerriAI/litellm/issues/3893#issuecomment-2549080296

* feat(langfuse_prompt_management.py): working e2e langfuse prompt management

works with `langfuse/` route

* feat(main.py): initial tracing for dynamic langfuse params

allows admin to specify langfuse keys by model in model_list

* feat(main.py): support passing langfuse credentials dynamically

* fix(langfuse_prompt_management.py): create langfuse client based on dynamic callback params

allows dynamic langfuse params to work

* fix: fix linting errors

* docs(prompt_management.md): refactor docs for sdk + proxy prompt management tutorial

* docs(prompt_management.md): cleanup doc

* docs: cleanup topnav

* docs(prompt_management.md): update docs to be easier to use

* fix: remove unused imports

* docs(prompt_management.md): add architectural overview doc

* fix(litellm_logging.py): fix dynamic param passing

* fix(langfuse_prompt_management.py): fix linting errors

* fix: fix linting errors

* fix: use typing_extensions for typealias to ensure python3.8 compatibility

* test: use stream_options in test to account for tiktoken diff

* fix: improve import error message, and check run test earlier
2024-12-20 13:30:16 -08:00
Ishaan Jaff
7959dc9db3
(feat) add bedrock/stability.stable-image-ultra-v1:0 (#6723)
* add stability.stable-image-ultra-v1:0

* add pricing for stability.stable-image-ultra-v1:0

* fix test_supports_response_schema

* ci/cd run again
2024-11-14 14:47:15 -08:00
Krrish Dholakia
3560f0ef2c refactor: move all testing to top-level of repo
Closes https://github.com/BerriAI/litellm/issues/486
2024-09-28 21:08:14 -07:00
Renamed from litellm/tests/test_cost_calc.py (Browse further)