* test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs
* test: restore lost coverage in litellm_utils offline replacements
Bound live health-check tasks to the concurrency limit so eager task
creation behind a semaphore fails, route the Langfuse trace-id check
through completion -> Logging.get_trace_id with all four metadata
combinations, run the callback dedup checks through acompletion, pass
the dynamic key via litellm_params, cover the env-inferred default
model list, and add the per-provider audio transcription config lookup
* test: restore router coverage lost in the offline move
Add the missing non-stream LIT-3058 header/count-once test, pin the UTC minute on every
usage-counter test, assert router-level client reuse for transcription, and replace the
private selector-attribute checks with routing outcomes per strategy. Assert outbound bodies
for speech, rerank, image, assistants and moderation, add timeouts to event waits, and drop
the router_unit_tests husk files that no longer hold tests
* test: restore sync stream, router sync stream, error event and field-type coverage for migrated responses tests
Add the sync streaming logging and sync Router.responses streaming cases the
offline replacements dropped, port the legacy per-event and response field-type
validation, assert raw headers, the float created_at conversion, MCP tool headers,
the search_context_size input, and add an offline replacement for the in-stream
context-window error event. Remove helpers left dead in the legacy file.
* test: run numpydoc-backed unit tests in GHA and drop a test that pinned a crash
* test: restore sequence number, item id and content part checks in the responses stream validator
* test: match router logging events to the test's own deployment so late events from other tests are ignored
* test: restore the s3 cold storage history test to its original assertions
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: delete 77 legacy tests covered by e2e, unable to fail, or dead in CI
* test: keep router helper tests and coverage ignore list unchanged
* test: keep cohere error handling tests
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move the unit half of 126 mixed legacy files into tests/unit
* test: restore litellm globals that moved tests set
* test: finalize migration test cleanup
* test: restore original bodies of moved legacy tests
The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit
test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated
* test: drop module imports shadowed by restored local imports
* test: assert on LiteLLM output in no-assertion moved tests and isolate leaks
Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change
Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags
* test: drain queued logging tasks before the Phoenix span test
The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: delete unconditionally skipped legacy tests
* test: move whole-unit legacy test files into tests/unit
* test: keep moved legacy tests free of import-time global state
* test: keep the module-level invocation scan pointed at tests/local_testing
* ci: drop the agent_testing CircleCI job emptied by the move
* test: fix moved-test isolation and router coverage
* test: add Tinyfish search package marker
* test: isolate moved tests from logger state leaks
* test: isolate Helicone logging fixture state
* test: isolate Vertex pass-through credentials between moved tests
* test: cancel S3 periodic flush tasks started by moved tests
* ci: restore CircleCI assistant test selection after move
* test: prevent Datadog datetime import shadowing
* ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move
* test: deduplicate imports in rebased unit tests
* test: remove duplicate passthrough router patch import
* test: remove moved legacy source files after rebase
* test: align moved tests with rebased main
* test: carry main's legacy-file edits into moved destinations
* test: make the moved cost map fallback tests assert the fetch and the backup
The four fallback cases only checked the result was non-empty, so they still
passed with integrity validation disabled. They now inject a mock client, assert
one fetch happened, and assert the result is exactly the local backup with the
fallback reason recorded.
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(exceptions): keep upstream 402 status and cool down 402 deployments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(exceptions): map 402 to PaymentRequiredError subclass of BadRequestError
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(exceptions): annotate PaymentRequiredError methods
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): skip 402 cooldown on single-deployment model groups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(exceptions): single prefix and 402 fallback response for PaymentRequiredError
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): map billing_error to PaymentRequiredError regardless of status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): honor explicit allowed-fails policy for single-deployment 402s
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(anthropic): drop billing_error body mapping to PaymentRequiredError
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(exceptions): type the PaymentRequiredError constructor parameters
* test(integration): cover 402 PaymentRequiredError mapping and cooldown
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts
The Langfuse redaction unit test exports to a local OTLP capture instead of
polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill
test only requires spend rows for requests the surviving worker served. The
spend-routes sweep treats the intentional /spend/capture_rate 503 as expected.
CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper
instead of reading it as "variable exists". The integration egress guard now
matches the script's own cgroup, so it no longer blocks the CircleCI agent,
which runs as the same user.
* fix(ci): keep the policy-load backoff typed as float
* fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests
* fix(secrets): load CyberArk policy one request at a time per manager
* test(secrets): pin that non-conflict CyberArk policy failures are not retried
* test(unit): run tests/unit with only an allowlisted host environment
CircleCI's unit job inherits every project env var, so real provider keys,
REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that
assume none are set. Locally, litellm's import-time load_dotenv did the same
from any .env up the tree. The unit conftest now drops every variable outside
a small allowlist and disables dotenv before litellm is imported.
* test(e2e): name a failed search and the stuck batch status instead of misattributing them
The websearch session test read an empty web_search_tool_result_error block as a
successful search, so a failing search tool surfaced as a session billing bug.
The batch cancellation timeout now reports the last status the proxy returned.
* fix(ci): scrub the host environment per unit test instead of for the whole pytest process
GHA shards run tests/unit next to other suites in one process, so the import-time
scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP
upstream fell back to the SDK2 interpreter. The two websearch tests that called
OpenAI and Perplexity live are removed: tests/unit no longer sees their keys.
* fix(ci): scrub only the host variables present before litellm is imported
The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to
its bundled encodings, so tokenizer paths tried to download them and hit the
socket guard. The prisma setup test now passes its own database URL instead of
reading one another test leaked into the process environment.
* fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue
LoggingWorker marked a task done on whichever queue was current when the callback
finished, so a callback that outlived an event-loop change raised "task_done()
called too many times" or undercounted the new loop's queue. It now settles the
queue the task came from.
The rest are test isolation fixes for failures that only appeared when another
file ran first on the same xdist worker: a replaced user_api_key_cache, breaker
metrics unregistered by prometheus tests, semantic_router's health-check filter on
uvicorn.access, logging tasks carried over from bedrock tests, a Router-written
model_cost entry, and a stray post captured by the langflow test. The token
counter check now asserts bounded chunking instead of wall-clock time.
* test(e2e/ui): wait for the logout redirect before visiting a protected page
Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED).
* test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests
* test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read
* test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect
* test(integration/mcp): answer the model-info probe another test's proxy sends to the model double
* fix(router): match provider-prefixed fallback keys for bare model groups
* fix(router): infer the fallback key's provider the way routing does for bare model groups
A bare model group served by a wildcard deployment (claude-sonnet-4-6 routed to anthropic/*) now finds a fallback keyed <provider>/<group>. The provider is inferred through one shared helper, inferred_provider, which the pattern router already used inline, so the fallback lookup and routing agree on the prefix. The lookup only infers a provider when some fallback key ends in /<group>, so alias-style groups never hit the resolver
* fix(router): resolve context window and content policy fallback keys through the shared lookup
---------
Co-authored-by: Jason Dougherty <jasondoc3@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(vertex): native batch JSONL passthrough with cost tracking
Add a per-request `passthrough=true` multipart field on `POST /v1/files`
(and the same kwarg on `litellm.create_file`) that uploads a native
Vertex AI batch JSONL to the deployment's GCS bucket unchanged, so rows
using `googleSearch` and other Gemini-only features run as written and
the output, `groundingMetadata` included, comes back untouched.
Passthrough is sticky through the GCS object path
(`litellm-vertex-files/passthrough/...`), so batch create and output
retrieval inherit it without new state. Native output rows are costed
from their `usageMetadata` with the deployment's model and model_info,
in the polling and retrieve paths and for the existing global
`disable_vertex_batch_output_transformation` flag, which billed $0
before.
The proxy requires the target to resolve to vertex_ai deployments only,
refuses `passthrough` with a non-batch purpose, a non-default
`target_storage`, or pre-call guardrails, and validates native rows on
`request` instead of the OpenAI batch keys.
* refactor(vertex): keep native batch row pricing inside the Vertex adapter
Moves native Vertex batch row detection, response parsing, and per-row
pricing from litellm/batches/batch_utils.py into
litellm/llms/vertex_ai/batches/transformation.py, so batch_utils only
aggregates the rows it gets back. Adds tests/test_litellm/files to the
misc unit shard so the new test directory is claimed by a shard.
* fix(files): say what a passthrough batch upload takes when a row is not native
The missing-key 400 listed bare key names, so an OpenAI-shaped row under
passthrough=true read "Each line must be a JSON object with keys request".
The batch line shape now carries its own hint, and the passthrough one says
a passthrough upload takes native Vertex batch rows with a request key
* fix(batches): bill native Vertex embedding batch rows on the native cost path
A native Vertex output row whose response holds an embedding was validated as a
generateContent response, so the documented tokenCount-only shape counted as a failed
row. Price embedding rows from their own usage (promptTokenCount, else tokenCount) with
the helper the transformed embeddings path already used, and drop the prompt-details
helper nothing calls anymore.
* fix(batches): keep modality batch rates on native Vertex embedding rows
An embedding row that carries usageMetadata was billed from promptTokenCount alone, so
its promptTokensDetails no longer reached the audio, image, and video batch rates the
way it did before the native cost path. Run every row with usageMetadata through the
Gemini usage parser and keep the flat tokenCount fallback for embedding rows without it.
* fix(batches): price native Vertex batch rows by modelVersion under a wildcard deployment
A `vertex_ai/*` deployment hands the batch cost path `*` as the deployment model, which
no cost map resolves, so every native (passthrough or flag-on) row was billed at $0. A
wildcard deployment model now defers to the row's own `modelVersion`, the way the
transformed path already prices by the row's `model`.
Also moves the native passthrough tests under tests/test_litellm, the tree codecov
reads, and covers the raw upload chunking, the embedding output translation, the
unpriceable-row path, and the flag-on dispatch.
* fix(batches): keep explicit deployment prices for native Vertex rows without a modelVersion
Under a wildcard deployment a native batch row that carries no modelVersion (an embedding
row, or a generateContent row Vertex returned without one) was billed at $0 even when the
deployment's model_info sets explicit batch prices, because the cost calculator was never
called. The row now falls back to the wildcard name, which the cost calculator prices from
the explicit model_info, and only a row with neither a modelVersion nor a deployment model
is billed at $0 with the warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The router coverage gate wants every router.py function reached by name from
a router test. The two per-endpoint attempt functions were only reached through
their callers, so each now has a direct test proving the per-request controls
carrier never reaches the provider call and every hop's stream comes back wrapped.
The /v1/responses and /v1/messages streaming wrappers only ever wrapped the
primary's stream, so a hop reached through the regular fallback chain had no
mid-stream handler: its failure re-raised, or the outer wrapper retried the
same entry with a fresh attempted set and never reached the rest of the list.
Every attempt of the chain now runs through a per-endpoint attempt function
that wraps its own stream, mirroring chat completions, and the per-request
fallback and retry overrides ride a frozen carrier so each hop's re-entry
still sees them after the retry layer pops them.
Offload the per-block hashing through offload_token_count on both the pre-call
read and the success-event write, hash raw bytes as base64 instead of raising,
drop the unused serialize_object helper, and bind the chained digest, the
message envelope, and the bytes path in the regression tests
A stream counted before its usage is known increments TPM by zero, so the
worker that served it never refreshed its local TPM value from Redis and
the first byte headers reported the token count another worker had already
consumed. Both pipeline operations now always run, matching the pre-change
callback, so the returned values refresh both worker local keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Router.make_call now increments the deployment TPM/RPM counter before set_response_headers reads remaining usage, so the headers carry post-increment values directly and the in-flight subtraction workaround from LIT-2719 is removed. deployment_callback_on_success adds only the tokens not yet counted (streams) and never a second request. The counter key uses the resolved deployment name so wildcard routes are read back correctly, and the proxy strips the router-owned counted-tokens marker from client metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_handle_clientside_credential registered the per-request Deployment it built for
a client-supplied api_key/api_base via upsert_deployment, which added it to
self.model_list under the shared model_name. That made a request-scoped
credential a permanent, load-balanced deployment that any later caller of the
same model group could be routed onto, reaching the provider with someone
else's forwarded credential.
The per-request Deployment still gets its own stable id for cooldown and
logging identity; it is just never registered with the router.
Resolves LIT-7811
The router clamps a negative request_retry_count found in request metadata before counting a failure, and the proxy strips a client-supplied request_retry_count with the other router-reserved metadata fields. The rust OCR lifecycle test that trips the per-request cap now plants request_retry_count instead of attempted_retries, which the cap no longer reads since the previous commit
num_retries_per_request has always capped the retries of one request with its fallback hops included. #40930 started reading the per-hop attempted_retries counter instead, and every fallback hop restarts that counter at zero, so a request could spend a fresh retry budget on each hop and the legacy fallback cap test started seeing the hop run.
Router.log_retry now also keeps request_retry_count on the request metadata, incremented on every retry and fallback hop and never truncated the way previous_models is, and max_retries_per_request_hit reads that count. The flat retry records, the litellm_metadata coverage and caps above four from #40930 stay as they are, and the legacy test goes back to its previous_models == 0 assertion.
Router.log_retry used to copy the failed attempt's kwargs and metadata into
metadata.previous_models. Nothing downstream read those copies, but they carried
client credentials into spend logs and grew the payload on every retry. Each
attempt now leaves a flat record (model group, deployment id, exception type and
string, attempt number), which drops RETRY_BREADCRUMB_EXCLUDED_KWARGS and the
per-retry credential masking.
num_retries_per_request was enforced from len(previous_models), which only
looked at the metadata bucket and never exceeded four records. The sync and
async client wrappers and the Rust lifecycle guard now read attempted_retries
from whichever metadata bucket the call carries.
Resolves LIT-7505
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cooldown entries rode the router-wide DualCache, which re-reads a key that is
missing from memory at most once every 10s. A deployment benched on one replica
therefore kept taking traffic on its siblings for up to 10 seconds, and the same
shared in-memory tier could evict a live cooldown once 200 unrelated router keys
crowded it out, which sent even the benching replica back to the dead deployment.
CooldownCache now owns a DualCache over the router's Redis with a 1s read
interval and an in-memory tier that only holds cooldown keys. Redis is attached
lazily because the router builds the cooldown cache before it wires Redis up.
test_no_linear_scans_in_router: #39674 renamed heuristic_v2_router_limit_violation
to auto_router_capability_violation, so the allowlist entry stopped matching and the
same admin-only scan tripped the static check. Rename the entry to follow it.
tableScrolling.spec.ts: 9ba6cab889 (LIT-4738) gave the Tags and Model Hub tables
client-side pagination at 25 rows, so the 40 seeded rows no longer render on one
page. Select 50 rows per page before counting, as the Logs case already does.
test_no_linear_scans_in_router: #39468 added config_deployments() and
heuristic_v2_router_limit_violation(), which both scan the whole model_list
from admin-only paths (model add/upsert), so add them to the allowlist. The
allowlist becomes a mapping so each exemption carries its reason as data.
test_missing_model_parameter_curl: a request with no model is rejected by the
proxy when nothing can serve it and by the router when a wildcard or default
deployment exists, and by the upstream provider when a wildcard forwards it,
so the message text is not a stable contract. Assert the contract that holds
in every case: HTTP 400 with a non-empty error message.
test_model_group_info_e2e: /model_group/info resolves wildcards, so it can
never return "anthropic/*" verbatim. cc3f9cd65b rewrote the assertion to
expect the raw pattern after claude-3-5-haiku-20241022 left the price map,
which made it unsatisfiable. Assert the expansion instead.
test_should_derive_ocr_mapping_status_from_live_tests: the audit needs a
native bridge built with the trace-parity feature, which CI never builds, so
skip with the harness's own diagnostic instead of erroring. Extract that
check out of ensure_trace_bridge as trace_bridge_error so a pytest run
reports the state without kicking off a maturin rebuild.
The Router executor only routed a query embedding when the vector store
carried extra embedding configuration, so a store registered with no
embedding model at all always went to the Router and 500'd on the
s3_vectors default text-embedding-3-small when no deployment served it.
Route on whether the Router serves the model, which is the rule the
executor had before, and keep the request metadata on the SDK fallback so
the embedding stays attributed either way.
Milvus REST and Azure AI Search still embedded the query through the SDK, so
a bare Router alias as litellm_embedding_model kept failing after the executor
landed for Valkey. Both now share BaseQueryEmbeddingVectorStoreConfig, which
embeds through the injected executor, drops the empty litellm_embedding_config
requirement, and awaits aembedding on the async path.
The Router executor falls back to the SDK for models the Router does not
serve, so inline provider configs such as azure/text-embedding-3-large with
their own credentials keep working through the proxy.
Tests fake OpenAI and Milvus at the HTTP boundary with respx instead of
patching litellm.embedding.
* fix(search): forward search-tool params through the router, complete Parallel AI v1 param mapping
SearchAPIRouter dropped every parameter configured on a search tool, forwarding
only per-request kwargs. Any tool-level setting (mode, max_results, ...) was
silently lost on the way to the adapter, for every search provider.
Also completes the Parallel AI v1 search surface: after_date, fetch_policy,
location and include_domains now nest under advanced_settings instead of being
sent as unknown top-level fields, responses preserve search_id / session_id /
warnings / raw excerpts, and search cost is derived from the request mode and
the provider's reported usage rather than a single flat rate.
* fix(parallel_ai): stop a caller from pricing its own search request
`_parallel_ai_usage` carries the provider's reported usage into cost
calculation. It was only written when the response contained a usage block, so
a caller could pass `_parallel_ai_usage=[{"name": "sku_search", "count": 0}]`
and, whenever the provider omitted usage, bill $0.00 instead of $0.005 — the
value also reached the upstream request body as an unknown field.
The key is now stripped from inbound params and written unconditionally from
the parsed response, so only the provider can populate it.
* fix(parallel_ai): price fast search mode correctly
* test(parallel_ai): fake search at HTTP boundary
* fix(parallel_ai): tolerate null search result fields
---------
Co-authored-by: khushishelat <shelatkhushi@gmail.com>
* fix(router): resolve fallbacks against the tier a pre-routing hook selected
A complexity or auto router picks a tier behind the router group name, but
fallback lookup kept using kwargs["model"], which is still the router name. The
tier's configured chain never ran, so a provider failure on its first hop went
straight back to the client with "No fallback model group found for original
model_group=smart-router".
The hook assigns the selected model to a local only, and fallback resolution runs
on an outer kwargs dict that **kwargs already copied, so writing it there is not
visible. Record the selection in the metadata bucket instead, which is a nested
dict shared by reference across those copies and is how the router already
carries values back up, then key fallback lookup off it when present.
Applies to the generic, context-window, content-policy and weighted-failover
lookups. Reporting keeps using the router name, since that is what the caller
asked for.
Fixes#38832
* fix(router): annotate the recorded-selection helper with a read-only mapping
record_pre_routing_selection only reads the request kwargs, writing into the
nested metadata bucket it finds there, so Mapping states what it actually needs
and clears the LIT001 mutable-annotation budget without a suppression.
* test(router): assert the no-kwargs path leaks nothing
The tolerated-None case called the helper without checking anything, which the
test-quality gate counts as a test with no assertion. Assert that a fresh mapping
still reads back empty, so the case proves the call is a no-op rather than only
that it does not raise.
* fix(router): stop declaring loop-assigned locals Final in the selection helpers
Both helpers annotated a loop-assigned local as Final, which reassigns a Final on
every iteration and cost three basedpyright errors. Read the buckets through a
generator instead, so the write path iterates a for-target and the read path
resolves in one shot with next(), which also matches the functional style the
type-discipline rules ask for.
* style(router): apply ruff format to the selection helpers
* fix(router): derive the pre-routing tier fresh on every fallback hop
The metadata buckets also carry whatever the caller sent, so an inbound
pre_routing_selected_model let a client pick which fallback chain its
request fell into. A fallback hop also inherited the previous hop's tier,
so the second hop keyed its own failure off the tier that already failed
and never ran its own chain.
Clear the key at the top of async_function_with_fallbacks. Every hop
re-enters there, so only the hook that routed that hop can set it.
* fix(router): drop the cast at the fallback-hop clear call site
* feat(router): fall back on anthropic safeguard refusals on /v1/messages
---------
Co-authored-by: Priyansh Nandwana <nandwana.priyansh103@gmail.com>
Container create and list requests had no container ID to decode, so the
router called the provider handler directly and the OpenAI transformation
fell back to the global OPENAI_API_KEY. Proxies configured only with
model_list credentials sent Authorization: Bearer None. Route through
_ageneric_api_call_with_fallbacks when the caller passes a model, expose
the list endpoint's model query param to the router, and encode the
managed container ID on the async create path so follow-up calls route
to the same deployment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
