litellm/tests/documentation_tests
ryan-crabbe-berri 4af59d7c6e
ci: lint the test tree for undefined names and fix all 30 (#37671)
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.

Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:

- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
  on the failure path, so the NameError, not the assertion, is what ran.
  test_llm_guard_error_raising is the worst: it passes today with content
  safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
  supports 3.10. Guarded behind the exceptiongroup backport that anyio already
  pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
  a helper that catches HTTPException by a name it never imported, so the
  challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
  module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
  test_claude_agent_sdk, and an unused helper in test_end_users calling a
  function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
  FileNotFoundError behind a NameError.

Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
2026-08-20 13:30:34 -07:00
..
test_api_docs.py Support budget/rate limit tiers for keys (#7429) 2024-12-26 19:05:27 -08:00
test_circular_imports.py Litellm remove circular imports (#7232) 2024-12-14 16:28:34 -08:00
test_env_keys.py fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996) 2026-08-05 14:49:55 -07:00
test_exception_types.py [Fix] Scope documentation workflow to match CircleCI and add missing router settings 2026-03-28 11:23:53 -07:00
test_general_setting_keys.py [Fix] Scope documentation workflow to match CircleCI and add missing router settings 2026-03-28 11:23:53 -07:00
test_optional_params.py Litellm dev 12 28 2024 p1 (#7463) 2024-12-28 20:26:00 -08:00
test_readme_providers.py feat(vector_stores): add Valkey as a managed vector store provider (#37002) 2026-08-18 21:45:22 +00:00
test_requests_lib_usage.py LITELLM: Remove requests library usage (#7235) 2024-12-17 12:50:04 -08:00
test_router_settings.py ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
test_standard_logging_payload.py [Fix] Scope documentation workflow to match CircleCI and add missing router settings 2026-03-28 11:23:53 -07:00