Commit graph

60 commits

Author SHA1 Message Date
ryan-crabbe-berri
8298cbe1f7 test: gate the test tree on B003 so a test cannot swap os.environ for a plain dict 2026-08-25 15:43:26 -07:00
yuneng-jiang
6a0d03914c
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite

TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.

Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.

Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.

* test: drop the duplicate imports the sys.path sweep exposed to F811

* test(pre-call-utils): restore the os import the new bedrock tests need
2026-08-22 09:25:58 -07:00
ryan-crabbe-berri
243ed4393d test: reject assertions on a caught error inside except (ruff PT017)
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.

Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
2026-08-21 13:35:08 -07:00
ryan-crabbe-berri
b76def0e5d
test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.

PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.

PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
2026-08-20 20:24:49 -07:00
ryan-crabbe-berri
680bcfd8aa
test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731)
* test(lint): ban blind pytest.raises(Exception) with ruff B017

A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.

All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.

Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.

* fix(test): widen the openai batch-dispatch assertion to OpenAIError

The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
2026-08-20 18:09:42 -07:00
Mateo Wang
9f97111edd
feat(fireworks_ai): sync chat completions endpoint with full API surface (#30885)
* feat(fireworks_ai): sync chat completions endpoint with full API surface

Add 23 missing request parameters to get_supported_openai_params():
seed, top_logprobs, min_p, typical_p, repetition_penalty,
mirostat_target, mirostat_lr, logit_bias, echo, echo_last, ignore_eos,
prompt_cache_key, prompt_cache_isolation_key, raw_output,
perf_metrics_in_response, return_token_ids, safe_tokenization,
service_tier, metadata, speculation, prediction, stream_options,
sampling_mask. Also add reasoning_history gated on supports_reasoning.

Fix prompt_truncate_length to prompt_truncate_len to match the actual
API parameter name. The old name was never in
DEFAULT_CHAT_COMPLETION_PARAM_VALUES, so it always went to extra_body
and was rejected by Fireworks; it never actually worked.

Normalize reasoning_effort boolean values to strings: True becomes
"medium", False becomes "none". The Fireworks OpenAPI schema
documents these as accepted types, but the server rejects non-string
values with HTTP 400 in practice. Integers pass through as-is since the
server is expected to validate them.

Auto-inject stream_options.include_usage=true when stream=true and the
user has not explicitly set stream_options. Without this, Fireworks
returns null usage in all streaming chunks, which is inconsistent with
the non-streaming behavior where usage is always present. If the user
explicitly sets include_usage=false, it is preserved.

Capture Fireworks-specific response fields in transform_response():
perf_metrics, prompt_token_ids, raw_output, and token_ids are now
extracted from the response and stored in response._hidden_params
(fireworks_perf_metrics, fireworks_prompt_token_ids,
fireworks_raw_outputs, fireworks_token_ids) so they are accessible to
logging, the proxy, and downstream consumers when the corresponding
request parameters are enabled.

Remove deprecated document inlining logic. Document inlining was
deprecated on 2025-06-30
(https://docs.fireworks.ai/updates/changelog#-document-inlining-deprecation).
This removes _add_transform_inline_image_block(), the file-to-image_url
migration in _transform_messages_helper(), and the
disable_add_transform_inline_image_block lookup. Current models that
support image input do so natively as VLMs. cache_control,
provider_specific_fields, and thinking_blocks stripping is retained.

Update get_provider_info() to look up supports_vision and
supports_pdf_input from the model cost map instead of hardcoding both
to True (which was based on the now-deprecated document inlining).
supports_prompt_caching remains True.

API docs: https://docs.fireworks.ai/api-reference/post-chatcompletions
Reasoning guide: https://docs.fireworks.ai/guides/reasoning
Prompt caching: https://docs.fireworks.ai/guides/prompt-caching

* fix fireworks chat api surface gaps

* Scope Fireworks thinking param to reasoning models

* style: fix black formatting

* fix(test): update minimax-m3 expected_vision to True

* test: cover non-dict content branch in transform_messages_helper

* fix(fireworks_ai): remove metadata from supported params to prevent internal metadata disclosure

* test(fireworks_ai): replace stale document-inlining capability test

The CircleCI-only litellm_utils_tests suite still asserted the old
behavior where document inlining made every Fireworks model report
supports_pdf_input and supports_vision as True. That premise was removed
in this change, so the test now reflects cost-map-driven capabilities:
unmapped models no longer advertise vision/PDF support while mapped VLMs
like minimax-m3 still do.

* test(fireworks_ai): add end-to-end regression for native OpenAI params

The existing coverage for the newly supported OpenAI-native params asserted
list membership in get_supported_openai_params or called map_openai_params
with a hand-built dict, both of which bypass the get_optional_params gate
(DEFAULT_CHAT_COMPLETION_PARAM_VALUES). That gate is what previously raised
UnsupportedParamsError for seed, top_logprobs, logit_bias, prompt_cache_key,
service_tier and prediction when drop_params=False. Assert the full path so a
revert of the supported-params additions fails the test instead of passing a
shallow membership check.

* test(fireworks_ai): fix test isolation in vision/inlining tests

Use monkeypatch in test_fireworks_ai_vision_capability_from_cost_map so the
LITELLM_LOCAL_MODEL_COST_MAP env var and litellm.model_cost are restored after
the test instead of leaking global state into the rest of the process.

Switch the document-inlining integration tests off deepseek-v3p1, whose
supports_vision is null in the cost map, onto minimax-m3 which is explicitly
supports_vision:true. The pass-through assertions no longer depend on a model
incidentally not being marked non-vision.

* fix(fireworks_ai): gate image rejection on exact vision capability

The image_url rejection read supports_vision via _get_model_cost_capability,
which falls back to hyphen-boundary substring matching when no exact cost-map
entry exists. A custom or fine-tuned model id that merely contains a known
non-vision model's short name (e.g. an id ending in -glm-5p2) inherited that
entry's supports_vision:false and hard-failed valid image_url blocks on a
vision-capable deployment.

Split the exact candidate-key lookup into _get_model_cost_capability_exact and
use it for the hard rejection so a fuzzy match can never block images; the
substring fallback stays a soft signal for capability reporting. Also rewrites
the fallback as a comprehension + max instead of an accumulating loop.

* feat(fireworks_ai): surface response fields on streaming responses

The Fireworks-specific response fields (perf_metrics, prompt_token_ids,
per-choice raw_output and token_ids) were only captured into _hidden_params in
transform_response, which runs for non-streaming completions; streaming chat
went through the default OpenAI chunk handler and dropped them.

Add a FireworksAIChatCompletionStreamingHandler that the provider now returns
from get_model_response_iterator. It reuses one extraction helper with
transform_response and attaches the fields to each streamed chunk's
provider_specific_fields, which is the channel litellm preserves when it
rebuilds streamed chunks (per-chunk _hidden_params is not carried through).
Per-choice token_ids/raw_output ride the content chunks; response-level
perf_metrics/prompt_token_ids ride the final usage chunk. Covered by an
end-to-end streaming test through litellm.completion(stream=True).

---------

Co-authored-by: Ahmad Shahzad <ahmad@shahzad.dev>
Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>
2026-06-20 19:49:07 -07:00
Mateo Wang
2c733c00f5
chore(ci): modernize model references in tests and configs (#27856)
* test: modernize models used in CircleCI e2e test suites

Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.

- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
  still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
  gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
  (also aligning oai_misc_config model_name with what
  test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
  -> claude-sonnet-4-5-20250929

* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5

Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.

* test: modernize models across remaining CI-mounted configs & tests

Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).

Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
  gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
  gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
  text-embedding-ada-002 underlying to text-embedding-3-small. User-
  facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
  preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
  bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
  claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
  haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.

Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
  + paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
  and test_bedrock_anthropic_messages_test.py: bump router fixtures
  using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
  modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.

* test: modernize placeholder model literals in router_unit_tests

Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.

Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
  claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
  claude-sonnet-4-5-20250929 / claude-opus-4-7 /
  claude-haiku-4-5-20251001 as appropriate

Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers

* test: modernize placeholder model literals across remaining CI suites

Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.

Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
  is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
  / gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro

Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
  translation/transformation logic). Only the deprecated 20250514
  references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
  documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
  upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
  match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
  equivalent).
- Top-level tests calling the proxy through user-facing aliases
  (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
  in proxy_server_config.yaml stay; only the underlying model was
  bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
  is model-name handling).
- Fake / mock / openai/fake identifiers.

Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
  spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
  resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
  (bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
  kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.

* test: fix CI failures from model modernization sweep

CI surfaced 4 categories of regression from the bulk modernization:

1. Azure deployment names are customer-specific. Reverted:
   - tests/litellm_utils_tests/test_health_check.py: azure/text-
     embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
     account does not have a text-embedding-3-small deployment).
   - tests/logging_callback_tests/test_custom_callback_router.py:
     same revert for two router fixtures driving aembedding.

2. gpt-5 family does not accept temperature != 1. Tests that pass a
   custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
   non-reasoning OpenAI mini that still accepts temperature/logprobs):
   - tests/logging_callback_tests/test_datadog.py
   - tests/logging_callback_tests/test_langsmith_unit_test.py
   - tests/logging_callback_tests/test_otel_logging.py

3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
   gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
   tests/test_openai_endpoints.py::test_chat_completion_streaming
   exercises logprobs/top_logprobs through that alias. Bumped the
   underlying model to gpt-4.1 (non-reasoning, still modern).

4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
   pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
   hardcoded model="gpt-4o" and a model-specific spend value. Reverted
   the litellm.acompletion calls in the test to model="gpt-4o" so the
   fixture's exact-match assertions still hold.

5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
   anthropic.messages.create routing to openai/gpt-5-mini returned an
   empty content[0] with max_tokens=100 (reasoning-token consumption).
   Swapped to openai/gpt-4.1-mini.

* test: fix Assistants API model + 2 cursor[bot] review nits

1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
   isn't accepted by the /v1/assistants endpoint
   ("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
   API-supported, non-reasoning).

2. example_config_yaml/pass_through_config.yaml: the previous sweep
   bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
   tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
   Sonnet tier intact. (Cursor bugbot review.)

3. example_config_yaml/simple_config.yaml: model_name was left as
   gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
   muddles the "simple" example. Make both sides gpt-5-mini so the
   most basic example is a straight 1:1 mapping again. (Cursor bugbot
   review.)

* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models

tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
2026-05-15 15:44:28 -07:00
user
7497674661 fix(proxy): sanitize redaction controls at ingress 2026-04-29 22:52:31 -07:00
user
842eea0131 chore(proxy): harden request control fields 2026-04-29 22:35:17 -07:00
Ryan Crabbe
dc46467235
fix(tests): replace deprecated Bedrock Claude 3.7 Sonnet model ID
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`,
returning 404s with "This model version has reached the end of its life."
Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability
surface: thinking, tools, prompt caching, PDF input, vision, computer use).

The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5
is converse-only on Bedrock.
2026-04-28 14:24:19 -07:00
Ishaan Jaffer
e8461b5b97
style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
yuneng-jiang
c9f7075690 Replace additional deprecated models across test files
- tests/local_testing/test_completion_cost.py:
  - claude-3-5-sonnet-20240620 -> claude-sonnet-4-6
  - gemini/gemini-1.5-flash-001 -> gemini/gemini-2.5-flash

- tests/test_litellm/test_utils.py:
  - claude-3-5-sonnet-20240620 -> claude-sonnet-4-6 (VertexAI config test, proxy tests)
  - gemini-1.5-pro -> gemini-2.5-pro (pre_process_non_default_params)
  - gemini/gemini-1.5-pro -> gemini/gemini-2.5-pro (proxy tests)

- tests/litellm_utils_tests/test_utils.py:
  - claude-3-opus-20240229 -> claude-sonnet-4-6 (trimming, vision tests)
  - gemini-pro -> gemini-2.5-pro (function calling test)
  - gemini-pro-vision -> gemini-2.5-flash (vision test)
  - gemini-1.5-pro -> gemini-2.5-pro (response schema test)
  - gemini/gemini-1.5-flash -> gemini/gemini-2.5-flash (function calling test)
  - gemini-1.5-pro -> gemini-2.5-pro (vision gemini test)
  - gpt-4-vision-preview -> gpt-4o (vision test)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 17:03:54 -07:00
Ishaan Jaff
afdf70be73
fix(test): update claude model name in test_get_valid_models_from_dynamic_api_key (#21771) 2026-02-21 10:15:23 -08:00
Tomu Hirata
43ba7d0a07 Add test case for Databricks Meta LLaMA 3.1 70B instruct model in content parsing tests
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
2026-02-17 15:36:00 +09:00
Yuta Saito
898cc3ff4f test: update langfuse trace_id tests to use litellm_trace_id 2026-01-22 06:19:43 +09:00
Sameer Kankute
e315e4cd7c fix litellm/tests/litellm_utils_tests/test_utils.py 2026-01-21 18:50:02 +05:30
Ishaan Jaffer
eaa7e61f57 test fixes 2025-12-05 17:12:01 -08:00
Colin Lin
0bd144103d [stripe] simplify opus test 2025-12-05 10:56:09 -05:00
Colin Lin
3046b9f163 [stripe] opus budget thinking 2025-12-05 10:56:02 -05:00
Cesar Garcia
5e70c78b94
fix(cost-tracking): support base_model lookup in litellm_metadata for Responses API (#16778)
Cost tracking was failing for Responses API when using custom deployment names
with base_model configuration. The issue occurred because:

- Chat Completions API stores model_info in 'metadata'
- Responses API stores model_info in 'litellm_metadata'
- Cost calculator only checked 'metadata', missing Responses API costs

Changes:
- Updated _get_base_model_from_metadata() to check both metadata locations
- Added comprehensive unit tests covering all scenarios
- Maintains backward compatibility (metadata takes precedence)

Fixes #16772
2025-11-18 19:53:18 -08:00
Ishaan Jaffer
95b1608970 test_get_valid_models_with_custom_llm_provider 2025-11-15 09:43:10 -08:00
Ishaan Jaffer
94c2c28f3d claude-sonnet-4-5-20250929 fix 2025-10-31 18:20:52 -07:00
Ishaan Jaffer
ae7b13550e test_models_by_provider 2025-10-23 09:10:41 -07:00
Ishaan Jaffer
8cb66168bc test fix 2025-10-10 19:57:17 -07:00
Georg Wölflein
dbfa8ec921
Fix end user cost tracking in the responses API (#15124)
#13860
2025-10-02 15:13:57 -07:00
Krish Dholakia
d4540d31c1
Merge branch 'main' into fix/streaming-tool-call-indices 2025-09-21 21:24:22 -07:00
Ishaan Jaffer
c6afa904bb fix: test_completion_with_no_model 2025-09-18 10:17:09 -07:00
Ishaan Jaffer
1e1d174733 fix: test_completion_with_no_model 2025-09-18 10:13:32 -07:00
Tim Elfrink
c5ca2afec3 Add test for tool call sequential index assignment
- Test multiple tool calls without explicit indices receive sequential indices
- Verify Delta class assigns indices 0, 1, 2... instead of defaulting all to 0
- Add comprehensive assertions for tool call details preservation
- Cover provider-agnostic streaming response scenarios
2025-09-15 21:11:13 +02:00
Krrish Dholakia
d05f58721e test: remove end of life model from tests 2025-09-09 21:01:45 -07:00
Ishaan Jaff
d37be48a80 test: llama-3.3-70b-versatile 2025-09-01 20:14:12 -07:00
Krish Dholakia
3e764ec268
Merge pull request #13808 from mainred/validate_api_version
feat(utils.py): accept 'api_version' as param for validate_environment
2025-08-22 23:59:38 -07:00
Ishaan Jaff
e93e266f84
[Performance] Use O(1) Set lookups for model routing (#13879)
* o(1) lookups

* Revert "o(1) lookups"

This reverts commit 620d142469.

* o(1) lookups

* Revert "o(1) lookups"

This reverts commit 676a9f5bcc.

* o(1) lookups

* register_model fix

* test_aget_valid_models

* lambda ai models fix

* test_utils.py

* test fix vertex ai
2025-08-21 22:56:46 -07:00
Qingchuan Hao
f2a6be390b feat(utils.py): accept 'api_version' as param for validate_environment 2025-08-20 14:29:58 +00:00
Krrish Dholakia
f544a4e238 test: update test 2025-07-29 21:08:36 -07:00
Robert Gambee
52b2984792
[Bug Fix] Always include tool calls in output of trim_messages (#11517)
* Check content and order of trimmed messages

* Assert tool calls are preserved if below max_tokens

* Unreverse order of tool calls

* Return tool calls alongside other messages

* Write test for trimming untokenizable field

* Return original messages in case of exception
2025-07-17 16:01:59 -07:00
Ishaan Jaff
c31a7d3ab7 fix new utils tests 2025-07-04 18:30:50 -07:00
Ishaan Jaff
6b623f9c98 test whitelisted models 2025-06-28 14:46:16 -07:00
Bougou Nisou
58dda44fda
feat: enhance redaction functionality for EmbeddingResponse (#12088) 2025-06-27 21:30:26 -07:00
Ishaan Jaff
7d47417906 test: fixes 2025-05-31 12:42:56 -07:00
Ishaan Jaff
0590b1eb3a
[Fix] Prometheus Metrics - Do not track end_user by default + expose flag to enable tracking end_user on prometheus (#11192)
* fix: testing for disabling end user on metrics

* fix: fixes for test_prometheus_factory

* Delete litellm/model_prices_and_context_window_backup.json

* fix: issues with merge conflicts

* fix: test_get_end_user_id_for_cost_tracking_prometheus_only

* Update tests/test_litellm/integrations/test_prometheus.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-05-27 17:06:58 -07:00
Ishaan Jaff
86cdb8382b
[Feat] Use aiohttp transport by default - 97% lower median latency (#11097)
* fix: add flag for disabling use_aiohttp_transport

* feat: add _create_async_transport

* feat: fixes for transport

* add httpx-aiohttp

* feat: fixes for transport

* refactor: fixes for transport

* build: fix deps

* fixes: test fixes

* fix: ensure aiohttp does not auto set content type

* test: test fixes

* feat: add LiteLLMAiohttpTransport

* fix: fixes for responses API handling

* test: fixes for responses API handling

* test: fixes for responses API handling

* feat: fixes for transport

* fix: base embedding handler

* test: test_async_http_handler_force_ipv4

* test: fix failing deepeval test

* fix: add YARL for bedrock urls

* fix: issues with transport

* fix: comment out linting issues

* test fix

* test: XAI is unstable

* test: fixes for using respx

* test: XAI fixes

* test: XAI fixes

* test: infinity testing fixes

* docs(config_settings.md): document param

* test: test_openai_image_edit_litellm_sdk

* test: remove deprecated test

* bump respx==0.22.0

* test: test_xai_message_name_filtering

* test: fix anthropic test after bumping httpx

* use n 4 for mapped tests (#11109)

* fix: use 1 session per event loop

* test: test_client_session_helper

* fix: linting error

* fix: resolving GET requests on httpx 0.28.1

* test fixes proxy unit tests

* fix: add ssl verify settings

* fix: proxy unit tests

* fix: refactor

* tests: basic unit tests for aiohttp transports

* tests: fixes xai

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2025-05-23 22:55:35 -07:00
Krrish Dholakia
33814917fe fix(token_counter.py): handle empty lists 2025-05-02 08:07:08 -07:00
Carlos Freund
cb177dbd7a
Fix and rewrite of token_counter (#10409)
* added tests

messages_with_counts: Made tolerance explicit for each test. But they match the new implementation(which beats the old)

* new token counter impl

* compare old and new implementation in test

* delete old token counter

* moved tests to /tests/litellm/litellm_core_utils

* use existing types

* docstrings

* warn about using default params on unknown model.

* created type for the token_counter_function

* check key == "content"

* throw error on invalid detail-type, ignore type-warning.

* fix imports
2025-05-01 23:34:37 -07:00
Ruperto A. Martinez
298a3574f4
Add supports_pdf_input: true to Claude 3.7 bedrock models (#9917)
* Add supports_pdf_input: true to Claude 3.7 bedrock models

* update unit test

---------

Co-authored-by: RupertoXTI <rmartinez@xtillion.com>
2025-05-01 14:56:54 -07:00
Krish Dholakia
33ead69c0a
Support checking provider /models endpoints on proxy /v1/models endpoint (#9958)
* feat(utils.py): support global flag for 'check_provider_endpoints'

enables setting this for `/models` on proxy

* feat(utils.py): add caching to 'get_valid_models'

Prevents checking endpoint repeatedly

* fix(utils.py): ensure mutations don't impact cached results

* test(test_utils.py): add unit test to confirm cache invalidation logic

* feat(utils.py): get_valid_models - support passing litellm params dynamically

Allows for checking endpoints based on received credentials

* test: update test

* feat(model_checks.py): pass router credentials to get_valid_models - ensures it checks correct credentials

* refactor(utils.py): refactor for simpler functions

* fix: fix linting errors

* fix(utils.py): fix test

* fix(utils.py): set valid providers to custom_llm_provider, if given

* test: update test

* fix: fix ruff check error
2025-04-14 23:23:20 -07:00
Ishaan Jaff
f9ce754817
[Feat] Add litellm.supports_reasoning() util to track if an llm supports reasoning (#9923)
* add supports_reasoning for xai models

* add "supports_reasoning": true for o1 series models

* add supports_reasoning util

* add litellm.supports_reasoning

* add supports reasoning for claude 3-7 models

* add deepseek as supports reasoning

* test_supports_reasoning

* add supports reasoning to model group info

* add supports_reasoning

* docs supports reasoning

* fix supports_reasoning test

* "supports_reasoning": false,

* fix test

* supports_reasoning
2025-04-11 17:56:04 -07:00
Krish Dholakia
ccbac691e5
Support discovering gemini, anthropic, xai models by calling their /v1/model endpoint (#9530)
* fix: initial commit for adding provider model discovery to gemini

* feat(gemini/): add model discovery for gemini/ route

* docs(set_keys.md): update docs to show you can check available gemini models as well

* feat(anthropic/): add model discovery for anthropic api key

* feat(xai/): add model discovery for XAI

enables checking what models an xai key can call

* ci: bump ci config yml

* fix(topaz/common_utils.py): fix linting error

* fix: fix linting error for python38
2025-03-27 22:50:48 -07:00
Krish Dholakia
c0845fec1f
Add OpenAI gpt-4o-transcribe support (#9517)
* refactor: introduce new transformation config for gpt-4o-transcribe models

* refactor: expose new transformation configs for audio transcription

* ci: fix config yml

* feat(openai/transcriptions): support provider config transformation on openai audio transcriptions

allows gpt-4o and whisper audio transformation to work as expected

* refactor: migrate fireworks ai + deepgram to new transform request pattern

* feat(openai/): working support for gpt-4o-audio-transcribe

* build(model_prices_and_context_window.json): add gpt-4o-transcribe to model cost map

* build(model_prices_and_context_window.json): specify what endpoints are supported for `/audio/transcriptions`

* fix(get_supported_openai_params.py): fix return

* refactor(deepgram/): migrate unit test to deepgram handler

* refactor: cleanup unused imports

* fix(get_supported_openai_params.py): fix linting error

* test: update test
2025-03-26 23:10:25 -07:00
Ishaan Jaff
1d7accce9e test_supports_web_search 2025-03-22 13:49:35 -07:00