`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
Python binds a name once per scope, so when a module or class defines the same
test twice only the last one exists. The earlier definitions are unreachable:
pytest never collects them, and nothing that references them can fail.
A sweep in August cleared nine of these. Five have appeared since, which is the
argument for a rule rather than another sweep.
Each survivor is the better version, so nothing is lost. The two SQS logger
twins additionally stub `asyncio.create_task`, which the shadowed copies did
not. The cost-calculator duplicate is a two-line stub that also takes a
`model_item` parameter no fixture supplies, so it could not have run even
unshadowed. The two `test_prompt_caching` bodies are both `pass`.
Collecting the four files reports 416 tests before and after.
`tests/proxy_unit_tests/conftest copy.py` goes with them. pytest only loads a
file named exactly `conftest.py`, nothing imports this one, and the space in the
name says what it was.
test_completion_fireworks_ai and test_completion_cost_fireworks_ai
made real Fireworks calls and broke whenever Fireworks rotated its
serverless catalog (no externally-verifiable model list exists).
They also asserted nothing — just printed.
Mock the HTTP post and assert real behavior instead: the request is
built with the right model/messages and the OpenAI-compatible
response parses back; the cost path yields a non-zero cost against
the local cost map. No network, no model dependency, stronger than
the old smoke checks.
Fireworks removed llama-v3p3-70b-instruct from serverless, so every
live test using it now fails with NotFoundError ("Model not found,
inaccessible, and/or not deployed").
Swap the 6 references (3 files) to the currently-served
accounts/fireworks/models/deepseek-v3p1 — the canonical model in
Fireworks' current docs examples and present in LiteLLM's cost map.
test_get_model_params_fireworks_ai is a pure pricing-heuristic test
(no network) asserting the >16b branch, so it uses llama-v3p1-70b-
instruct instead to keep the "fireworks-ai-above-16b" assertion and
branch coverage intact.
The test_chat_completion_low_budget test was flaky because async spend
tracking couldn't reliably catch up within 50 calls with 0.5s sleeps.
Increased to 200 calls with 0.1s sleeps (same total time budget) to
give more opportunities for budget enforcement to trigger.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The recent commit 2a997993d4 replaced httpx.AsyncClient() with
get_async_httpx_client() in ui_sso.py, but the PKCE tests still
patched the old httpx.AsyncClient path. Updated all 10 affected
tests to mock get_async_httpx_client and removed unnecessary
context manager setup since AsyncHTTPHandler is returned directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
gemini/gemini-2.5-flash lacks cache_creation_input_token_cost in the
model cost map, causing a TypeError when the test multiplies
cache_creation_input_tokens by None. Use claude-haiku-4-5 instead,
which has the required prompt caching cost fields.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix intent params
* Add responses
* fix unrelated test
* test fix - fireworks API endpoint is down
* test fix fireworks ai is having an active outage
* test_completion_cost_databricks
* dbrx fix test API currently not responding
* Update OpenAI Realtime handler to use the correct endpoint and include all query parameters. Adjusted error messages for missing API base and key. Updated health check URL construction to pass model as a query parameter.
* Enhance OpenAI Realtime handler tests to ensure model parameter inclusion in WebSocket URL. Added new tests to verify correct URL construction with model and additional parameters, preventing 'missing_model' errors. Updated existing tests for consistency.
* Remove debug print statements for API base and key in OpenAIRealtime handler to clean up the code.
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* fix(exception_mapping_utils.py): correctly pass through 504 status code
openai also raises a 504 status code
* build(model_prices_and_context_window.json): add gpt-4o-mini-tts to model cost map
Fixes https://github.com/BerriAI/litellm/issues/9591
* fix(cost_calculator.py): fix input cost calculation for gpt-4o-mini-tts
Fixes https://github.com/BerriAI/litellm/issues/9591
* test: testing updates
* fix(cost_calculator.py): handle custom pricing at deployment level for router
* test: add unit tests
* fix(router.py): show custom pricing on UI
check correct model str
* fix: fix linting error
* docs(custom_pricing.md): clarify custom pricing for proxy
Fixes https://github.com/BerriAI/litellm/issues/8573#issuecomment-2790420740
* test: update code qa test
* fix: cleanup traceback
* fix: handle litellm param custom pricing
* test: update test
* fix(cost_calculator.py): add router model id to list of potential model names
* fix(cost_calculator.py): fix router model id check
* fix: router.py - maintain older model registry approach
* fix: fix ruff check
* fix(router.py): router get deployment info
add custom values to mapped dict
* test: update test
* fix(utils.py): update only if value is non-null
* test: add unit test
* fix(vertex_and_google_ai_studio_gemini.py): log gemini audio tokens in usage object
enables accurate cost tracking
* refactor(vertex_ai/cost_calculator.py): refactor 128k+ token cost calculation to only run if model info has it
Google has moved away from this for gemini-2.0 models
* refactor(vertex_ai/cost_calculator.py): migrate to usage object for more flexible data passthrough
* fix(llm_cost_calc/utils.py): support audio token cost tracking in generic cost per token
enables vertex ai cost tracking to work with audio tokens
* fix(llm_cost_calc/utils.py): default to total prompt tokens if text tokens field not set
* refactor(llm_cost_calc/utils.py): move openai cost tracking to generic cost per token
more consistent behaviour across providers
* test: add unit test for gemini audio token cost calculation
* ci: bump ci config
* test: fix test
* fix(core_helpers.py): handle litellm_metadata instead of 'metadata'
* feat(batches/): ensure batches logs are written to db
makes batches response dict compatible
* fix(cost_calculator.py): handle batch response being a dictionary
* fix(batches/main.py): modify retrieve endpoints to use @client decorator
enables logging to work on retrieve call
* fix(batches/main.py): fix retrieve batch response type to be 'dict' compatible
* fix(spend_tracking_utils.py): send unique uuid for retrieve batch call type
create batch and retrieve batch share the same id
* fix(spend_tracking_utils.py): prevent duplicate retrieve batch calls from being double counted
* refactor(batches/): refactor cost tracking for batches - do it on retrieve, and within the established litellm_logging pipeline
ensures cost is always logged to db
* fix: fix linting errors
* fix: fix linting error
* feat(bedrock/rerank): infer model region if model given as arn
* test: add unit testing to ensure bedrock region name inferred from arn on rerank
* feat(bedrock/rerank/transformation.py): include search units for bedrock rerank result
Resolves https://github.com/BerriAI/litellm/issues/7258#issuecomment-2671557137
* test(test_bedrock_completion.py): add testing for bedrock cohere rerank
* feat(cost_calculator.py): refactor rerank cost tracking to support bedrock cost tracking
* build(model_prices_and_context_window.json): add amazon.rerank model to model cost map
* fix(cost_calculator.py): bedrock/common_utils.py
get base model from model w/ arn -> handles rerank model
* build(model_prices_and_context_window.json): add bedrock cohere rerank pricing
* feat(bedrock/rerank): migrate bedrock config to basererank config
* Revert "feat(bedrock/rerank): migrate bedrock config to basererank config"
This reverts commit 84fae1f167.
* test: add testing to ensure large doc / queries are correctly counted
* Revert "test: add testing to ensure large doc / queries are correctly counted"
This reverts commit 4337f1657e.
* fix(migrate-jina-ai-to-rerank-config): enables cost tracking
* refactor(jina_ai/): finish migrating jina ai to base rerank config
enables cost tracking
* fix(jina_ai/rerank): e2e jina ai rerank cost tracking
* fix: cleanup dead code
* fix: fix python3.8 compatibility error
* test: fix test
* test: add e2e testing for azure ai rerank
* fix: fix linting error
* test: mark cohere as flaky
* test(test_completion_cost.py): add unit testing to ensure all bedrock models with region name have cost tracked
* feat: initial script to get bedrock pricing from amazon api
ensures bedrock pricing is accurate
* build(model_prices_and_context_window.json): correct bedrock model prices based on api check
ensures accurate bedrock pricing
* ci(config.yml): add bedrock pricing check to ci/cd
ensures litellm always maintains up-to-date pricing for bedrock models
* ci(config.yml): add beautiful soup to ci/cd
* test: bump groq model
* test: fix test
* fix(utils.py): initial commit fixing custom cost tracking
refactors out provider specific model info from `get_model_info` - this was causing custom costs to be registered incorrectly
* fix(utils.py): cleanup `_supports_factory` to check provider info, if model info is None
some providers support features like vision across all models
* fix(utils.py): refactor to use _supports_factory
* test: update testing
* fix: fix linting errors
* test: fix testing
* test(test_completion_cost.py): add sdk test to ensure base model is used for cost tracking
* test(test_completion_cost.py): add sdk test to ensure custom pricing works
* fix(main.py): add base model cost tracking support for embedding calls
Enables base model cost tracking for embedding calls when base model set as a litellm_param
* fix(litellm_logging.py): update logging object with litellm params - including base model, if given
ensures base model param is always tracked
* fix(main.py): fix linting errors
* fix(bedrock/converse_handler.py): fix bedrock region name on async calls
* fix(utils.py): fix split model handling
Fixes bedrock cost calculation when region name is given
* feat(_health_endpoints.py): support health checking datadog integration
Closes https://github.com/BerriAI/litellm/issues/7921
* fix(__init__.py): fix init to exclude pricing-only model cost values from real model names
prevents bad health checks on wildcard routes
* fix(get_llm_provider.py): fix to handle calling bedrock_converse models
* feat(cost_calculator.py): add cost tracking ($0) for openai moderations endpoint
removes sentry cost tracking errors caused by this
* build(teams.tsx): allow assigning teams to orgs
