scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 83
litellm/tests
History
michelligabriele 5dd77eb8ca
fix(openai/realtime): drop model from upstream URL for intent=transcription (#43854) 
* fix(openai/realtime): drop model from upstream URL for intent=transcription

* fix(openai/realtime): keep model on the upstream URL for non-OpenAI hosts

A custom api_base on the openai provider can be a gateway that routes on
the model query param, so the transcription model drop now applies only
to OpenAI's own hosts. xAI's URL is unchanged again

* test(integration): audit realtime transcription upstream URL on a custom api_base

Twenty-two integration cells cover transcription and conversation sessions on every realtime route, the OpenAI SDK sync and async clients, malformed and duplicated query params, unauthenticated and refused upgrades, an unknown model, idempotent spend logging, a twenty-session burst, an upstream outage mid burst, a worker kill, and a proxy restart, each asserting the exact query pairs the upstream received. The scripted upstream now records every websocket upgrade as an observation

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-05 19:14:56 +00:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
agent_tests
audio_tests test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests ci: move Postgres, MCP and Redis suites to CircleCI integration (#44453) 2026-10-05 09:33:14 -07:00
documentation_tests fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
e2e fix(mcp): bind OAuth clients to their upstream issuer (#37777) 2026-10-05 10:25:47 -07:00
guardrails_tests test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
image_gen_tests test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
integration fix(openai/realtime): drop model from upstream URL for intent=transcription (#43854) 2026-10-05 19:14:56 +00:00
litellm_utils_tests test(ci): pin the ROI estimator flag and the prompt-cache counter in two drifted tests (#44499) 2026-10-04 08:40:23 -07:00
llm_responses_api_testing test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
llm_translation test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
load_tests
local_testing test: repair stale and polluting tests red on scheduled main CI (#44229) 2026-10-02 21:32:17 +00:00
logging_callback_tests test(ci): pin the ROI estimator flag and the prompt-cache counter in two drifted tests (#44499) 2026-10-04 08:40:23 -07:00
mcp_tests fix(mcp): preserve upstream tool schemas and parameter headers (#44425) 2026-10-03 15:00:43 -07:00
multi_instance_e2e_tests
ocr_tests refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
openai_endpoints_tests test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
otel_tests test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
pass_through_tests fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
pass_through_unit_tests feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
proxy_admin_ui_tests
proxy_behavior feat(lens): coordinate worker releases and bundled installs (#44428) 2026-10-03 16:33:45 -07:00
proxy_e2e_anthropic_messages_tests fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1 2026-09-19 23:33:29 +00:00
proxy_migration_tests fix(proxy-extras): bound the lock waits of the partitioned SpendLogs index build (#44109) 2026-10-01 18:28:50 -07:00
proxy_security_tests refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits 2026-09-19 18:53:14 -07:00
proxy_unit_tests ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
router_unit_tests fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
rust-python-harness refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
search_tests fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) 2026-09-25 19:12:36 -07:00
spend_tracking_tests test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
store_model_in_db_tests test(ci): fix six CircleCI test regressions on main (#44429) 2026-10-03 20:55:01 -07:00
test_litellm fix(tracing): preserve spend identity and gateway correlation (#44421) 2026-10-03 16:00:49 -07:00
test_litellm_rust feat(tracing)!: return only data from SQL queries (#44609) 2026-10-05 12:11:59 -07:00
unified_google_tests test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
unit fix(openai/realtime): drop model from upstream URL for intent=transcription (#43854) 2026-10-05 19:14:56 +00:00
vector_store_tests
windows_tests fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_fallbacks.py test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_new_vector_store_endpoints.py
test_openai_endpoints.py test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py test(proxy): delete the legacy proxy test tree and shard tests/unit/proxy by glob (#44018) 2026-10-01 11:40:45 -07:00
test_resource_cleanup.py
test_rust_python_harness.py test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) 2026-09-25 19:10:20 -07:00
test_service_logger_otel.py
test_spend_logs.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_team.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_team_logging.py
test_team_members.py test(proxy): move management_endpoints, management_helpers and guardrails tests into tests/unit/proxy (#44003) 2026-10-01 10:52:03 -07:00
test_users.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.