Commit graph

159 commits

Author SHA1 Message Date
yuneng-jiang
6a0d03914c
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite

TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.

Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.

Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.

* test: drop the duplicate imports the sys.path sweep exposed to F811

* test(pre-call-utils): restore the os import the new bedrock tests need
2026-08-22 09:25:58 -07:00
ryan-crabbe-berri
243ed4393d test: reject assertions on a caught error inside except (ruff PT017)
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.

Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
2026-08-21 13:35:08 -07:00
ryan-crabbe-berri
e9d40a8f73 test: enforce F811 so a duplicate definition cannot silently replace the first
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.

311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.

The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
2026-08-21 12:06:19 -07:00
yuneng-jiang
35416c702d
test: point the live together_ai suites at a model together still serves (#37746)
Every live together_ai call in CI has answered 503 Service unavailable since
2026-08-20, across two runs 2.5 hours apart, while Together's status page
reported no incident in either window. These are real calls, not replayed
cassettes: the VCR layer runs filter_non_2xx_response, so a 503 is never
written to a cassette and cannot be replayed back.

Qwen/Qwen2.5-7B-Instruct-Turbo does not appear anywhere on Together's monitored
component list, whose Qwen entries are all Qwen3.x, so a model-level outage
there would never surface as an incident. The same 503 already forced
test_basic_rerank_together_ai to be skipped on a different together_ai model,
so per-model 503s are an established failure mode here rather than a platform
outage.

openai/gpt-oss-20b is the cheapest together_ai entry that carries real pricing
and the capabilities these suites exercise, at $0.05/$0.20 per 1M tokens with
function calling, response schema and tool choice. Together monitors it as a
served component. The retired model also carries null pricing in the cost map,
which is its own liability now that unpriced models are blocked.

test_multiple_deployments.py keeps the old id: it is a router fallback list
that is green today, and busting its cassette to prove a point would trade a
passing test for a live call this change cannot vouch for.
2026-08-20 18:53:27 -07:00
ryan-crabbe-berri
4af59d7c6e
ci: lint the test tree for undefined names and fix all 30 (#37671)
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.

Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:

- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
  on the failure path, so the NameError, not the assertion, is what ran.
  test_llm_guard_error_raising is the worst: it passes today with content
  safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
  supports 3.10. Guarded behind the exceptiongroup backport that anyio already
  pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
  a helper that catches HTTPException by a name it never imported, so the
  challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
  module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
  test_claude_agent_sdk, and an unused helper in test_end_users calling a
  function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
  FileNotFoundError behind a NameError.

Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
2026-08-20 13:30:34 -07:00
Yuneng Jiang
075781568d
test: remove tests that never execute
Three groups, all verified by running the suite rather than by inspection.

18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.

30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.

9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.

One file that is a demo script rather than a test; its own docstring says to run
it with python.

Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
2026-08-12 10:45:38 -07:00
Yuneng Jiang
d49114b101
test(bedrock): repoint live Claude tests off the retired Claude 3 Sonnet
AWS no longer serves `anthropic.claude-3-sonnet-20240229-v1:0`. The streaming
path returns a plain 404, "Model with the provided id
anthropic.claude-3-sonnet-20240229-v1:0 is not found", and the non-streaming
path answers 500 for the same reason. Our own cost map has carried a
2026-07-30 deprecation date for it since #36538

That accounts for 20 failures across local_testing_part1, local_testing_part2
and llm_translation_testing. litellm maps both statuses correctly, so the
tests are what went stale, not the client

Replacement is `us.anthropic.claude-sonnet-4-5-20250929-v1:0`: a like-for-like
Sonnet, and the newest Bedrock Sonnet this repo exercises against the real API
in tests/e2e. Newer ids exist in the cost map, but nothing in the repo calls
them live, so picking one would be an unverified guess about model access on
the CI account

Scope is limited to the tests that actually issue a request. The occurrences
that assert on the model string itself, or that feed mocked transformations,
keep the old id so their assertions stay meaningful
2026-08-11 18:49:47 -07:00
Yuneng Jiang
b6557d2b14
test: repair three failing suites on litellm_internal_staging
The management route-coverage guard fires because /team/metadata_schema landed
in #33353 without a behavior-suite scenario, so this adds one covering the nine
seeded actors plus the unauthenticated 401

The prometheus budget-metric assertions read the log call's first positional
arg, which #35703 turned into an unrendered "%s" format string when it moved
logging to lazy args. They now render the message from the call args, which
also pins the arg order and the exception text that the old substring check
never reached

GitHub Models was fully retired on 2026-07-30, so test_completion_github_api
can no longer pass: the endpoint the github provider targets returns 404 and
models.github.ai answers 410 "github_models_retirement_brownout". The dead live
test is removed rather than skipped
2026-08-04 16:07:02 -07:00
Mateo Wang
b16cfd7de9
test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900)
* test: point router/completion/triton tests at the local fake OpenAI endpoint

The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down

Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream

* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI

* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target

* fix(tests): keep fake OpenAI mock alive across xdist workers

ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves
2026-06-20 16:20:35 -07:00
Mateo Wang
f11c12d157
Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326)
This reverts the Bedrock CI account migration (#28728). The original account
(888602223428) was put under an AWS security restriction after a leaked key
and has since been reactivated, while the replacement account (941277531214)
lacks access to several models the suites exercise (legacy Bedrock Claude 3
models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship
Opus). Pointing CI back at the reactivated account restores that coverage.

This is the exact inverse of #28728: all hardcoded 941277531214 references go
back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs
and their suffixes, batch execution role ARN, and the example proxy config),
the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs
revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge
Base revert to their original ids, and the live-call tests go back to the
legacy model strings. The grid_spec fail_reason workaround for the unentitled
Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field
added after the migration.

The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at
941277531214 and must be set to the reactivated account's fresh credentials
separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.
2026-05-30 11:26:24 -07:00
Mateo Wang
f9407bc036
chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)
* chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214

The original account (888602223428) was put under a security restriction by
AWS after a root access key leaked in a PR comment. While that account works
its way through the AWS Support unlock process, Bedrock-touching CI tests have
been migrated to a fresh account (941277531214).

Changes:
  - Replace 26 hardcoded references to 888602223428 with 941277531214 across
    8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime
    ARNs, batch execution role ARN, and example proxy config).
  - The provisioned-model and imported-model ARNs are referenced only from
    mocked unit tests — no AWS resources to recreate.
  - The batch execution IAM role has been recreated in the new account with
    the same name and equivalent permissions.
  - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC,
    hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account
    under the same names — see tools/agentcore-deploy/ in a follow-up.

CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME
were updated separately via the CircleCI API to point at the new account.

Smoke-tested locally against the new account:
  aws bedrock-runtime converse --region us-west-2 \
    --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \
    --messages '[{"role":"user","content":[{"text":"ping"}]}]'
  → 200, model returned 'pong'

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes

The first migration commit replaced just the account ID, but AgentCore
auto-assigns a random 10-char suffix to every runtime on creation — we
can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the
new account. Updated the AgentCore-runtime ARNs in the three files that
reference real runtime IDs (not the mock-based unit-test ARNs).

Deployed runtimes:
  arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp
  arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy

Both runtimes are status=READY and pass a smoke invoke:
  $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}'
  → 200, {"result": "echo: ping"}

The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the
deploy artifacts). Tests that only verify the SDK wiring will pass; if any
test asserts on agent output content, swap the echo for the real agent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(tests): point Bedrock batch tests at new-account S3 bucket

The account migration (888602223428 -> 941277531214) was a flat
account-ID swap, which only rewrites ARNs that embed the account
number. S3 bucket names carry no account ID, so the live Bedrock
batch tests still uploaded to `litellm-proxy` — a bucket that lives
in the old account. S3 names are globally unique, and the old account
still holds that name, so it can't be recreated in the new account.

Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees
global uniqueness). The bucket must be created in 941277531214 and the
batch execution role granted s3:GetObject/PutObject/ListBucket on it
before this job is run in CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(tests): point live S3 logging test at new-account bucket

Same account-ID-free blind spot as the batch bucket: `load-testing-oct`
lives in the old account and its name can't be reused globally. The
`logging_testing` CI job is wired into the workflow and runs
test_basic_s3_logging, which uploads to this bucket with the CI env
creds, then lists and deletes objects — a live dependency.

Rename to `load-testing-oct-941277531214`. The bucket must exist in the
new account with the CI IAM principal granted
s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(tests): repoint Bedrock guardrail IDs to new-account guardrails

The migration left guardrail IDs untouched (no account ID in them), so
all live guardrail tests failed with "guardrail identifier or version
does not exist" against 941277531214. Recreated both guardrails in the
new account and updated the hardcoded IDs:
  - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD,
    with explicit inputAction=ANONYMIZE so masking applies to INPUT,
    which is the source litellm's moderation hook sends)
  - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set
    to the exact string the tests assert on)

Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the
guardrailConfig in test_bedrock_completion.py. Verified locally: the 5
previously-failing guardrail tests now pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(bedrock): migrate legacy models to current inference profiles

The new CI account (941277531214) cannot invoke legacy Bedrock models
(AWS gates them: "marked by provider as Legacy... not actively using in
the last 30 days"). Migrated the live-call tests:
  - anthropic.claude-3-sonnet-20240229    -> us.anthropic.claude-sonnet-4-5-20250929-v1:0
  - anthropic.claude-3-haiku-20240307     -> us.anthropic.claude-haiku-4-5-20251001-v1:0
Current Claude models on Bedrock require the us. inference-profile prefix
(bare on-demand ids are rejected).

cohere.command-r-plus has no working replacement (all Cohere is legacy-
gated in the new account): swapped to claude-haiku-4-5 in provider-
agnostic param lists. amazon.titan-image-generator skipped (no working
replacement). Mocked/transformation/cost tests that reference the legacy
strings are intentionally left unchanged. Verified live against the new
account.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(bedrock): repoint SageMaker + Knowledge Base to new-account resources

These referenced account-scoped resources by hardcoded id that only
existed in the old account, so the migration's account-ID swap missed
them. Recreated in 941277531214 and repointed:
  - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614
    -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge)
  - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless
    vector store + titan-embed-text-v2, seeded with a LiteLLM doc)
Verified live: test_sagemaker.py (12 passed) and
test_bedrock_knowledgebase_hook.py (12 passed).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214)

claude-opus-4-7 is listed in the new Bedrock CI account's foundation
models but invoke is denied (AccessDeniedException: "not available for
this account"). Bedrock access to the flagship Opus requires an AWS
Sales request, not the self-serve model-access toggle, so it can't be
enabled inline with the rest of the account migration.

Add an optional `skip_reason` to ModelEntry and set it on the
bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip.
Cell count (231) and route coverage are unchanged, so the structural
asserts still pass. Restore coverage by deleting the one skip_reason
line once access is granted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(bedrock): swap/skip legacy-gated models unavailable on new CI account

The migrated AWS account (941277531214) cannot access several models that
the old account could, so the remaining red CI jobs were hitting real
Bedrock "Access denied / Legacy" and "account not authorized" errors:

- image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is
  legacy-gated), matching the existing titan skip.
- batches: skip test_async_file_and_batch (Bedrock batch inference is not
  authorized on the new account; requires an AWS support case).
- litellm_overhead: swap legacy claude-3-5-haiku for the active
  us.anthropic.claude-haiku-4-5 inference profile.
- test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the
  active us.anthropic.claude-sonnet-4-5 inference profile.

https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa

* test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account

- e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference
  is not authorized on account 941277531214) and migrate the missed
  s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214.
- build_and_test: swap legacy bedrock claude-3-sonnet for the active
  us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured
  output e2e test.

https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa

* test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (#28791)

Replace the silent skips added for the new CI account with noisier behavior:
- reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present)
  instead of skipping, so the missing entitlement stays visible in CI; they
  still skip when AWS creds are absent (local dev)
- Bedrock batch inference tests: drop the skip so they run and fail until
  batch access is granted
- Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the
  transform + cost-tracking path stays under test without live model access

https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT

Co-authored-by: Claude <noreply@anthropic.com>

* test(bedrock): use pytest.xfail for known-failing opus-4-7 cells

Replace pytest.fail with pytest.xfail when a model has a fail_reason,
so known-broken cells stay visible as XFAIL without keeping CI red.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

---------

Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
2026-05-25 12:03:17 -07:00
Yuneng Jiang
b5db7ed37d
test(fireworks): mock remaining live smoke tests
test_completion_fireworks_ai and test_completion_cost_fireworks_ai
made real Fireworks calls and broke whenever Fireworks rotated its
serverless catalog (no externally-verifiable model list exists).
They also asserted nothing — just printed.

Mock the HTTP post and assert real behavior instead: the request is
built with the right model/messages and the OpenAI-compatible
response parses back; the cost path yields a non-zero cost against
the local cost map. No network, no model dependency, stronger than
the old smoke checks.
2026-05-15 22:28:27 -07:00
Yuneng Jiang
39a1d438f2
test(fireworks): replace deprecated llama-v3p3-70b-instruct model
Fireworks removed llama-v3p3-70b-instruct from serverless, so every
live test using it now fails with NotFoundError ("Model not found,
inaccessible, and/or not deployed").

Swap the 6 references (3 files) to the currently-served
accounts/fireworks/models/deepseek-v3p1 — the canonical model in
Fireworks' current docs examples and present in LiteLLM's cost map.
test_get_model_params_fireworks_ai is a pure pricing-heuristic test
(no network) asserting the >16b branch, so it uses llama-v3p1-70b-
instruct instead to keep the "fireworks-ai-above-16b" assertion and
branch coverage intact.
2026-05-15 22:17:10 -07:00
Sameer Kankute
57eae8d01c
Merge branch 'litellm_internal_staging' into litellm_staging_03_22_2026
Some checks failed
Unit Tests: Caching (Redis) / caching-redis (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
2026-04-20 19:56:00 +05:30
Ishaan Jaffer
e8461b5b97
style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
Yuneng Jiang
95e1babf67
[Fix] TogetherAIConfig.get_supported_openai_params recursion
TogetherAIConfig.get_supported_openai_params called get_model_info(),
whose first line calls litellm.get_supported_openai_params() — which for
together_ai routes straight back into this method. The recursion only
terminated when Python's recursion limit was hit or when
_get_model_info_helper raised "not mapped" at the deepest level. Either
way the try/except caught it, so the bug stayed silent — but the cycle
ran ~332 deep every time, emitting hundreds of DEBUG log lines per
call. Surfaced as "infinite loop" in CI when the success_handler thread
emitted that log spam against an already-closed stderr during test
teardown.

Replace the get_model_info() call with supports_function_calling(),
which uses _get_model_info_helper directly and does not call
get_supported_openai_params. Measured drop from 332 to 2
_get_model_info_helper calls per first uncached lookup.

Also swap the test model from Qwen/Qwen3.5-9B (not in model_cost map)
back to a mapped serverless model, Qwen/Qwen2.5-7B-Instruct-Turbo. The
mapping gap is what made the recursion's tail end raise up into the
success handler during teardown in the first place.
2026-04-16 17:20:58 -07:00
Sameer Kankute
83d0f28aa8
Fix tests 2026-04-16 19:02:29 +05:30
Yuneng Jiang
a9c6156137
[Fix] Test - Together AI: replace deprecated Mixtral with serverless Qwen3.5-9B
Mixtral-8x7B-Instruct-v0.1 is no longer on Together AI's serverless tier
and now requires a dedicated endpoint, causing multiple tests to fail in CI:

  - test_together_ai.py::TestTogetherAI::test_empty_tools
  - test_completion.py::test_completion_together_ai_stream
  - test_completion.py::test_customprompt_together_ai
  - test_completion.py::test_completion_custom_provider_model_name
  - test_text_completion.py::test_async_text_completion_together_ai

Qwen/Qwen3.5-9B is currently serverless on Together AI and supports
function calling, satisfying BaseLLMChatTest capability requirements.
2026-04-14 17:43:35 -07:00
Ishaan Jaffer
a7bfe0c540 test_completion_azure 2026-03-30 21:07:49 -07:00
Ishaan Jaffer
98de60e741 test_completion_gpt4_turbo 2026-03-30 16:35:31 -07:00
Ishaan Jaffer
37bcc3252f test_completion_gpt4_turbo 2026-03-30 16:35:10 -07:00
yuneng-jiang
0ccaff8ed9
Merge pull request #24805 from BerriAI/litellm_/compassionate-kirch
[Test] Mock DeepInfra completion tests to avoid real API calls
2026-03-30 11:42:00 -07:00
Yuneng Jiang
868468db14
[Test] Mock DeepInfra completion tests to avoid real API calls
The DeepInfra tests were making real API calls and failing with AuthenticationError.
Mock the HTTP layer to verify request shape instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 11:07:34 -07:00
Krrish Dholakia
a92b31a636 test: remove dead tests 2026-03-28 20:36:08 -07:00
Krrish Dholakia
bc829d51f2 test: test 2026-03-28 19:17:38 -07:00
yuneng-jiang
82d3b23526 Update deprecated Together AI model in test_completion_together_ai_llama
Llama-3.2-3B-Instruct-Turbo is no longer available as a serverless model
on Together AI. Switch to Llama-3.3-70B-Instruct-Turbo which is still
available and has cost data in the model prices map.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 14:02:42 -07:00
yuneng-jiang
023654d9ad Fix flaky CI tests: mock timeout race, update deprecated model, fix callback leak
- test_hanging_request_azure: mock httpx.AsyncClient.send to simulate slow
  response instead of racing real network latency against a 10ms timeout.
  The old non-existent deployment (gpt-4o-new-test) returned 404 faster
  than the timeout, causing NotFoundError instead of APITimeoutError.
- test_completion_together_ai_llama: update model from deprecated
  Meta-Llama-3.1-8B-Instruct-Turbo to Llama-3.2-3B-Instruct-Turbo
  (Together AI removed the old model from serverless).
- conftest.py: clear litellm.callbacks list before each test to prevent
  proxy hooks (SkillsInjectionHook, VirtualKeyModelMaxBudgetLimiter)
  from leaking across tests via Router initialization.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 23:45:58 -07:00
yuneng-jiang
c27b65d09e [Fix] Replace deprecated claude-3-7-sonnet-20250219 with claude-sonnet-4-5-20250929 in test_completion
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-02-20 17:26:02 -08:00
Ishaan Jaffer
e5bbafba62 test_completion_openrouter_reasoning_effort 2026-02-14 12:04:05 -08:00
yuneng-jiang
c41459c8e3 fixing mistral model deprecation, cloud zero transform bug 2026-02-12 17:53:04 -08:00
shin-bot-litellm
fea40925cf
test: remove hosted_vllm from OpenAI client tests (#20163)
hosted_vllm no longer uses the OpenAI client, so these tests
that mock the OpenAI client are not applicable to hosted_vllm.

Removes hosted_vllm from:
- test_openai_compatible_custom_api_base
- test_openai_compatible_custom_api_video
2026-01-31 10:10:45 -08:00
Sameer Kankute
b729622bf5 Fix: generationConfig removal from tests 2026-01-22 19:00:37 +05:30
Ishaan Jaffer
5873cd4a32 test_completion_bedrock_titan_null_response 2026-01-07 15:09:44 +05:30
Ishaan Jaffer
269771a7c9 test_bedrock_httpx_streaming 2026-01-07 15:02:10 +05:30
Ishaan Jaffer
645ca64780 a121 fixes 2026-01-07 15:00:49 +05:30
Yuta Saito
23713d1811 fix: anthropic claude-3-opus-20240229 EOL 2026-01-06 16:06:21 +09:00
Ishaan Jaffer
148715541c test compl openai orgs 2025-12-20 23:41:29 +05:30
Ishaan Jaffer
6112160a16 Revert "[Fix] Security - Remove example API keys with high entropy (#18255)"
This reverts commit 24edbccf5c.
2025-12-20 20:48:11 +05:30
Alexsander Hamir
24edbccf5c
[Fix] Security - Remove example API keys with high entropy (#18255) 2025-12-19 10:09:50 -08:00
Alexsander Hamir
0c48826cdc
Revert "[Fix] CI/CD - local_testing & mapped tests (#18222)" (#18223)
This reverts commit dc7f500c47.
2025-12-18 14:46:09 -08:00
Alexsander Hamir
dc7f500c47
[Fix] CI/CD - local_testing & mapped tests (#18222) 2025-12-18 14:34:48 -08:00
Alexsander Hamir
6635325629
fix: filter internal params in fallback code and fix test issues (#17941)
- Filter skip_mcp_handler and other internal params in fallback_utils.py before calling acompletion
  Fixes issue where internal parameters were being passed to provider APIs causing errors
- Remove deployment field from GCS bucket logger test metadata
  Fixes model name mismatch where deployment field was overriding the model in logging
- Update Bedrock Titan test to use non-deprecated model (titan-text-express-v1)
  Fixes test failure due to deprecated amazon.titan-text-lite-v1 model
2025-12-13 15:05:26 -08:00
Ishaan Jaffer
01fd4d7cef fix fireworks test 2025-11-26 18:58:32 -08:00
Ishaan Jaffer
badbadba0d fix img URL for tests 2025-11-22 09:41:15 -08:00
Ishaan Jaff
bd7d653bae
Revert "Update perplexity cost tracking (#15743)" (#16345)
This reverts commit ad6a0f4d44.
2025-11-06 19:00:45 -08:00
Sameer Kankute
ad6a0f4d44
Update perplexity cost tracking (#15743)
* Update perplexity cost tracking

* fix lint errors

* fix code

* fix tests in perplexity

* fix test realted to api call

* fix exception test
2025-11-03 08:45:34 -08:00
Krish Dholakia
74ae7aed44
build: Squashed commit of the following: (#16176)
commit bb0b050fb0
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Sat Nov 1 20:00:01 2025 -0700

    test: update tests

commit b2da4bdac2
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Wed Oct 22 14:58:01 2025 -0700

    fix(langfuse_otel_attributes.py): log tools and other optional params

commit 75bee1f274
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Wed Oct 22 14:42:05 2025 -0700

    feat(langfuse_otel/): working request/response logging on spans

    Closes https://github.com/BerriAI/litellm/issues/13764

commit a3e4fa5b81
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Wed Oct 22 14:20:39 2025 -0700

    fix: initial commit fixing langfuse request/response logging with OTEL

commit 09fc9deac8
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Wed Oct 22 13:33:52 2025 -0700

    fix(litellm_logging.py): for responses api - return a unified usage object for logging

    ensures logging integrations all pull the right usage information
2025-11-02 09:46:40 -08:00
Ishaan Jaffer
2c52791b83 test_model_function_invoke 2025-10-25 14:56:12 -07:00
Ishaan Jaffer
e6b61213ca test_completion_azure_deployment_id 2025-10-25 12:26:06 -07:00
Ishaan Jaffer
762053a8e9 test_model_function_invoke 2025-10-25 12:26:06 -07:00