litellm/tests/code_coverage_tests
mika-lt d81c55f4cc
feat(custom_llm): let custom providers serve native generateContent (#51083)
* feat: support native generateContent in CustomLLM

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve custom generateContent accounting and built-in priority

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: forward custom generateContent success callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve custom generateContent failure callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: retain async callbacks across generateContent fallback

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(google_genai): scope custom generateContent hooks to native fields and stream only with a stream hook

* fix(google_genai): log custom adapter calls on the caller's logging object and reject untranslatable config

Custom adapter calls now share the caller's logging object, so the proxy's deferred logging covers them and one request logs one row under the caller's call id and call type. Native hook responses log from any pydantic, SDK, or dict shape, per-request prices apply to custom handlers, and a missing stream hook or a generationConfig key the adapter cannot send answers 400

* fix(google_genai): return custom hook objects as their camelCase dict so the router tags the deployment

* fix(google_genai): keep tool args, drop SDK-only fields, and let routers retry custom legs that cannot stream

Tool call args and function responses keep the caller's keys and null values. SDK objects no longer send sdkHttpResponse, automaticFunctionCallingHistory or parsed to the client or the logs. A custom leg without a stream hook answers 500 like CustomLLM's other unimplemented hooks, so a mixed router group retries the request on another leg. The wire-value walk is depth capped for the recursive function check

* fix(google_genai): keep router legs serving chat after a custom stream without a hook, and accept config the adapter already meets

* fix(google_genai): reject JSON mode a custom leg's completion adapter cannot send

The custom-leg generationConfig check now accepts responseMimeType and response schema keys only when the adapter sends them as response_format, or when they ask for plain text. JSON mode without an object schema answers 400 unless drop_params is set

* fix(google_genai): send a custom hook's dict back without SDK-only fields

A hook that returns a dumped google-genai response as a dict sent its snake_case keys, parsed, sdk_http_response and automatic_function_calling_history to the client, past the guardrail that rewrites only candidate text. Dicts now go back as the same filtered camelCase body as every other hook return, and the router tags that copy instead of writing into the hook's own dict

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 05:57:58 -05:00
..
azure_client_usage_test.py fix - correctly re-use azure openai client 2025-03-18 09:51:28 -07:00
ban_constant_numbers.py Squashed commit of the following: (#9709) 2025-04-02 21:24:54 -07:00
ban_copy_deepcopy_kwargs.py Fix - using managed files w/ OTEL + UI - add model group alias on UI (#13171) 2025-07-31 21:22:04 -07:00
bedrock_pricing.py Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_ruff_dead_test_code 2026-08-24 09:46:56 -07:00
callback_manager_test.py (Refactor / QA) - Use LoggingCallbackManager to append callbacks and ensure no duplicate callbacks are added (#8112) 2025-01-30 19:35:50 -08:00
check_data_replace_usage.py Bug fix - String data: stripped from entire content in streamed Gemini responses (#9070) 2025-03-07 21:06:39 -08:00
check_e2e_no_raw_requests.py test(e2e): share the coding-CLI harness between suites (#49370) 2026-11-02 13:50:56 -08:00
check_endpoint_coverage.py Revert "[Feature] Add /public/supported_endpoints endpoint" 2026-02-26 17:21:43 -08:00
check_fastuuid_usage.py code cov test script check_fastuuid_usage.py 2025-09-24 10:27:22 +09:00
check_get_model_cost_key_performance.py refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
check_guardrail_apply_decorator.py content filter test fix 2026-02-12 17:54:16 -08:00
check_licenses.py chore: move config and docs off the repo root (#50966) 2026-11-11 18:29:35 -08:00
check_migrations_lock_timeout.py chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
check_migrations_no_data_rewrites.py chore: move scripts and runtime sources under litellm (#51006) 2026-11-12 04:19:57 +00:00
check_no_none_string_error_fields.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
check_no_publicly_known_master_key.py fix(ui): rebuild the dashboard bundle and cover it in the example master key check (#47612) 2026-10-20 13:02:29 -07:00
check_prisma_binary_cache.py fix(ci): bound setup steps so pytest always gets its full budget 2026-08-10 16:22:07 +00:00
check_provider_folders_documented.py feat(harness): add pi harness for litellm.agent() (#45467) 2026-10-20 09:01:40 +00:00
check_py310_typing_imports.py fix: keep litellm importable on Python 3.10 and guard 3.11-only typing imports in CI (#39448) 2026-09-02 18:27:19 -07:00
check_spanattributes_value_usage.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
check_telemetry_imports.py feat(telemetry): add litellm.telemetry records, semantic group consent and sink protocol (#45484) 2026-10-13 18:19:22 -07:00
check_unbounded_in_lists.py ci: fail on new unbounded SQL IN lists and add a Prisma chunking helper (#42629) 2026-09-26 13:40:44 -07:00
check_unsafe_enterprise_import.py chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
check_workflow_job_name_collisions.py fix(ci): keep an expression matrix directive out of the comparison 2026-09-06 02:59:16 -07:00
check_workflow_startup_safety.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
code_qa_check_tests.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
dependency_license_cache.json chore: move config and docs off the repo root (#50966) 2026-11-11 18:29:35 -08:00
enforce_llms_folder_style.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
ensure_async_clients_test.py chore: move scripts and runtime sources under litellm (#51006) 2026-11-12 04:19:57 +00:00
info_log_check.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
liccheck.ini chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
license_cache.json fix(deps): update python-multipart to >=0.0.20 in CI and test configs 2026-03-03 15:10:39 -03:00
litellm_logging_code_coverage.py docs(litellm_logging_code_coverage.py): fix check 2025-06-18 21:36:03 -07:00
log.txt Squashed commit of the following: (#9709) 2025-04-02 21:24:54 -07:00
memory_test.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
prevent_key_leaks_in_exceptions.py fix(main.py): fix key leak error when unknown provider given (#8556) 2025-02-15 14:02:55 -08:00
recursive_detector.py feat(custom_llm): let custom providers serve native generateContent (#51083) 2026-11-14 05:57:58 -05:00
router_code_coverage.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
router_enforce_line_length.py Litellm router code coverage 3 (#6274) 2024-10-16 21:30:25 -07:00
router_litellm_globals_check.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
router_surface_allowlist.json refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
router_surface_check.py chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
test_aio_http_image_conversion.py fix img URL for tests 2025-11-22 09:41:15 -08:00
test_ban_set_verbose.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_chat_completion_imports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_check_no_none_string_error_fields.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_check_no_publicly_known_master_key.py chore: move ui and vscode-extension under litellm and delete docs (#50992) 2026-11-12 04:19:57 +00:00
test_e2e_idp_stack.py test(e2e): verify IdP readiness through real HTTP 2026-09-11 17:25:38 -07:00
test_e2e_junit_report.py test(mcp): replace the live Linear MCP OAuth e2e tests with integration coverage (#47154) 2026-10-17 14:29:59 -07:00
test_e2e_metadata.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_generate_provider_endpoints_support.py chore: move scripts and runtime sources under litellm (#51006) 2026-11-12 04:19:57 +00:00
test_merge_smoke.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_no_hardcoded_secrets.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_provider_cache.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_provider_replay_harness.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_proxy_types_import.py refactor(logging): load enterprise alerting loggers lazily so import litellm skips proxy types (#44762) 2026-10-05 22:06:53 -07:00
test_router_strategy_async.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_unit_passed_gate.py ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
test_workflow_job_name_collisions.py fix(ci): keep an expression matrix directive out of the comparison 2026-09-06 02:59:16 -07:00
unbounded_in_baseline.txt feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582) 2026-11-13 16:05:35 +00:00