scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-16 04:29:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 138
litellm/tests
History
devin-ai-integration[bot] 30ada9d26a
feat(batches): store each batch line's request and response in S3 and GCS (#41691) 
* feat(batches): emit per-line JSONL batch records to callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): only emit line items for final batches

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(batches): satisfy ruff format for line item logging

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): cover line-item edge shapes and general settings propagation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): keep deployment credentials out of callback-visible params

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(batches): declare credential transport attribute on Logging

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): integration contract for per-line batch callback events

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): assert per-line events beside the aggregate and create events

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): assert line item events carry no credentials

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): reset batch line-item flag when the setting is absent

Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): map anthropic batch lines through the anthropic message transformer

Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: tag the anthropic result cast with a cast-ok reason

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: shorten cast-ok reason so ruff format passes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): skip line-item callbacks for providers without a line reconstruction

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: wrap widened provider literals so ruff format passes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): pair bedrock lines by recordId and type them from the reconstructed response

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): keep line item events out of proxy limiters and strip credentials from child params

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): skip failed line item events in parallel request limiter failure hooks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): tolerate lifecycle spend rows when asserting no line item rows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): place mutable-ok tags on the lines the discipline gate flags

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): skip router budget accounting for batch line item events

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): keep anthropic line errors and skip native vertex batch rows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): restore test-quality tags on line item tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: move batch line-item unit test to tests/unit after merging main

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): emit batch line-item callbacks once per batch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): release line-item claim when fan-out emitted nothing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): make the line-item claim release ownership-safe

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): keep bodyless failed batch lines reaching callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(logging): seed batch line hidden params in get_hidden_params defaults

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(batches): reuse fetched batch result files for line-item logging

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): release line-item claim when nothing was emitted

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): type batch line-item callback sites to satisfy basedpyright budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: wrap batch hidden-params cast for ruff format

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): reason suppressions for batch line-item lint deltas

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: drop stale mutable-ok suppressions after LIT002 removal

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486)

* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters

Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id)
bypassed the call-type-only batch guards:

- ClickHouseSpendLogger wrote one spend row per line on top of the aggregate
  aretrieve_batch row, billing every batch twice with the flag on.
- Router deployment callbacks and the routing-strategy usage handlers charged the
  historical batch's tokens/rpm to the live TPM/RPM counters, so after
  Router.aretrieve_batch the next chat raised RouterRateLimitError.

Skip these events wherever the aggregate batch retrieve is already skipped.

* fix(logging): keep internal litellm_params authoritative in model_call_details

A request body key named litellm_params lands in optional_params, and
update_environment_variables merged **optional_params after the internal
entry, so a caller could overwrite the logging kwargs' litellm_params and
forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose
body carried {"litellm_params": {"batch_parent_id": ...}} returned 200
with no ClickHouse spend row. Re-assert the internal litellm_params after
the merge so every sink (ClickHouse, Postgres, router quota counters)
reads the same trusted value, and assert the aggregate aretrieve_batch
row still writes in the ClickHouse line-item test (review finding).

* fix(clickhouse): skip failed batch line items in the spend sink too

A failed JSONL line is part of the aggregate aretrieve_batch row's
request counts; writing it its own spend_logs row inflated request
counts with the line-item flag on. Extend the batch line-item guard to
the failure handler (review finding) and pin both directions: a failed
line item writes no row, a genuine live failure still does.

* fix(batches): declare batch_line_items service family on claim calls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(integrations): keep batch line items out of the built-in metering sinks

With store_batch_line_items_in_callbacks on, per-line batch events carry a
real response_cost and reached six metering sinks that lacked the
is_batch_line_item_event guard, so a completed batch metered aggregate +
per-line (roughly 2x true spend): prometheus (litellm_spend_metric and
request counters), openmeter, lago, datadog cost management (FOCUS
BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost
metrics. Skip line-item events in each sink's success/failure metering
entry points, matching the guards already on the Postgres and ClickHouse
spend sinks. OTel spans for line items are still emitted; only their
cost/token metrics are skipped. Tests drive real sink instances on the
real dispatch lists through a completed 2-line batch and assert each
sink meters exactly the aggregate.

* style: format metering-sinks test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test

Follow-ups from review on the sink guards: record_failure() now skips
batch line items too (a failed line's interval measures retrieval and
emission, not its model call), and the prometheus guard test snapshots
and restores the process-wide registry collectors instead of leaving
them unregistered. The OTel v2 test is now a flag on/off A/B over the
whole metric snapshot: cost and usage values must match exactly and no
metric may gain datapoints with the flag on.

* chore: finish main merge conflict resolution for batch line items

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: sort imports in litellm_logging after main merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(batches): use the public hook class names main now exports

* fix(batches): only the batch writer can mark a callback event as a line item

amoderation, transcription and speech build litellm_params from every request kwarg,
so a body field named batch_parent_id reached Logging.litellm_params and every
spend, budget and rate-limit hook skipped the request. update_environment_variables
now drops an incoming batch_parent_id; the line-item writer sets it through
Logging.mark_batch_line_item. Batch line hidden params are emitted only when set.

* chore(batches): keep the lint gates within main's ceilings after the merge

Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the
banned cast with a literal match, dropped isinstance checks main made redundant,
and moved test-quality reasons onto the lines the gate reads.

* fix(anthropic): read the message id off the validated message

The anthropic_messages logging path can hand over an already parsed message object,
which has no .get; main read its id through getattr, so take it from the validated
AnthropicResponse instead.

* fix(batches): an unreadable error file no longer drops the output line events

Costing counts an error file it could not fetch as no failed lines; line logging
refetched it and the raise aborted the whole fan-out, so the successful lines were
never emitted. The error file now reads as empty with a warning.

* test(batches): pin billing for batch_parent_id on every endpoint and value shape

Chat completions, embeddings, responses and messages join moderations, speech and
transcriptions, and speech is driven with an empty, int, list, object and 5 KB value.

* test(batches): bill every request of a concurrent burst mixing forged and plain bodies

* revert: keep the line-item fan-out all-or-nothing on an unreadable error file

85fc4498cc emitted the output lines and kept the batch claim when the error file
could not be read, so its failed lines were never retried. Raising releases the
claim and the next retrieve retries the whole fan-out.

* test(batches): pin the all-or-nothing retry and warm the burst model first

An unreadable error file emits no lines and the next retrieve emits all of them.
The burst cell drives one speech model after a warm-up request so a cold worker
cannot fail it, and waits for the batched key spend.

* fix(batches): run the line fan-out outside the aggregate event's deadline

The logging worker cancels a callback coroutine after its per-callback
deadline, and the line fan-out ran inside the aggregate aretrieve_batch event
before its callbacks, so a large batch or a slow sink could cancel the
aggregate billing event. The fan-out now runs in its own task, built from a
copy of the parent's litellm_params taken before the aggregate callbacks
write their request-local markers.

Anthropic keeps no input file and reports input_file_id "None", which the
fan-out fetched and failed on, so no Anthropic line ever arrived. That id is
now treated as no input file.

* fix(batches): let running line fan-outs finish before proxy shutdown closes their clients

A started fan-out keeps its batch claim, so a shutdown that cut it short
skipped the unsent lines until the claim expired. proxy_shutdown_event now
waits up to MAX_TIME_TO_CLEAR_QUEUE for them first. Also replace a test's
type: ignore with an isinstance check.

* fix(batches): keep batch line fields off callback kwargs and skip fan-outs at shutdown

Line bodies were spread into the callback kwargs, so a line's metadata could settle router rate limit reservations; they now stay in optional_params, and the router rate limit and prompt caching hooks skip line events.
A retrieve logged after the shutdown drain, or by the logging worker's exit flush, no longer starts a fan-out its loop would destroy while it holds the batch claim.
The output file is fetched before the error file again.

* fix(batches): log batch lines under the retrieve's callback settings and keep them out of alerting and shadow eval

Line events inherit the retrieve request's key and team callback settings, so turn_off_message_logging and per-team logging credentials apply to them, and they keep the system prompt in the logged messages.
Shadow eval, Datadog metrics, Slack daily reports and PagerDuty skip line events, since the batch API served them and the aggregate event already counts the batch.

* test(batches): line events follow the key's message logging opt-out

* test(batches): line events follow team logging settings, keep the line system prompt and survive a restart once

* fix(batches): keep in-memory line claims for 10,000 batches and pin managed batch emission across poller and retrieves

* Keep batch line unit tests in-process and drop repeated per-sink comments

* Share one batch provider alias across batch_utils and line item logging

* Report Bedrock line status codes, skip unknown Bedrock lines without raising, fetch the input file after the native Vertex check

* Read a swallowed Redis claim failure as unavailable, price unpriceable lines at zero, keep the OTel failure docstring

* Call handle_completed_batch when line items are off, keep a cache test from leaking Redis into the line claim cache, pin an unpriceable line's zero cost on the callback kwargs

* Hand the cost poller's credential snapshot to line item file reads, so a managed Bedrock batch's lines read its input from the deployment bucket

* Log batch lines through the sync callbacks a direct async call runs, keep failed lines out of router cooldown counts, and emit an error line whose body is text

* fix(batches): log failed Bedrock embedding lines as embeddings with their input

A Bedrock batch input record carries no url, so a failed Titan v2 line fell back to acompletion and its
inputText never reached the event messages. The failed line now takes aembedding from the deployment
model and the record's inputText is the line input

* fix(batches): keep a Bedrock line's system prompt in its logged messages

Bedrock batch input records hold the system prompt as content blocks, which logging only reads off the kwargs as text, so Claude and Nova lines dropped it

* feat(batches): store per-line batch requests and responses in S3/GCS via batch_line_item_callbacks

Replace the per-line logging path with a dedicated hook. When the hourly batch cost
poller finishes a managed batch, it pairs every output and error line with its input
line, prices each line, and hands one payload to every logger listed in
litellm_settings.batch_line_item_callbacks. s3_v2 and gcs_bucket write it as one
NDJSON object per batch under batch_line_items/<day>/<batch id>.jsonl.

Bodies are dropped and the header kept when the creator key or team, the global
setting, or the logger itself turns off message logging, and when the creator can
no longer be found. A failed upload is logged and never blocks spend tracking.

* fix(batches): store batch line items after the batch is billed and retired

Send a billed batch's lines only after its row is marked complete, so a slow or
failing sink never holds a claimed batch open. Skip reading the input file when
the bodies are dropped anyway, still send the error lines of a batch whose output
file is gone, and end every NDJSON record with a newline.

* test(batches): cover team-level and deleted-key redaction of batch line items

* fix(batches): store batch line items in the background so a hung bucket never holds up billing

The poller now retires a batch first and hands its line items to a per-worker background store that runs one batch at a time. A slow, hung or failing bucket no longer delays billing or retiring the other batches in the poll cycle. While a poll cycle's worth of batches is still waiting to store, the next batch's lines are dropped with an error.

* fix(batches): store an unbilled batch's line items only once it is retired

When retiring an expired, cancelled or all-failed batch fails, the batch stays on the poll page, so its lines are no longer read and stored again on every poll cycle until it is retired. The poller also releases a billed batch's output bytes before reading the next batch's output, as it did before line items existed.

* fix(batches): price and write batch line items off the event loop

Pricing every line of a large batch and serializing its object ran on the
proxy's event loop, so a 50k-line batch stalled the worker for seconds.
Both now run in a worker thread. An object the bucket refuses for good is
no longer kept in s3_v2's drop marks, which only a log-queue flush clears.
Vertex AI input rows carry the custom_id in their key or request labels, so
their lines now get the request too.

* fix(batches): pair no request for a shared custom_id and wait for an unclaimed batch's retry

A Vertex AI embeddings entry whose input is a list is split into one row per
item, all with the same custom_id, so its line now gets no request instead of
the last row. On a database without batch_processed nothing claims a billed
batch, so when marking it complete fails it stays in the poll page and its
lines now wait for the next poll instead of being written on every cycle.

* fix(batches): store batch line items only once the poll has billed every batch

Pricing a large batch's lines on a worker thread still competes with the poll for the GIL, so the next batches
in the same poll were billed and retired seconds later. The background stores now wait until the poll cycle that
queued them has finished.

* fix(batches): hash line-item objects off the event loop and keep lines with a lone surrogate

A batch's whole line-items object was hashed for its S3 PUT on the event loop, and a line
holding a lone surrogate from a provider's JSON escapes made the object impossible to
encode as UTF-8, so the batch's object was never written. The object is now prepared in a
worker thread, and only the line that cannot be encoded is written with escaped characters.

* fix(batches): keep request and response for batches created without a virtual key

A batch created with a JWT has no API key on its row, and its line items were stored without
request and response as if its creator had been deleted. Such a batch now follows its team's
and the proxy's message logging settings, the same as a master key batch. A deleted key or team
or a failed lookup still drops the bodies.

* fix(batches): pair line items with the right request on every deployment and skip empty objects

A batch retired with nothing billable now reads its input file by the id the provider reported, so a batch that
ran on the second deployment of a model group keeps each line's request. A Vertex AI line read as custom_id
"unknown" pairs with no input row, and a batch whose files hold no readable line writes no object

* fix(batches): let OAuth2 batches follow team and proxy message logging settings

The line-item redaction check looked up every saved creator credential other than the master key or a CLI session as a virtual key. An OAuth2 batch saves the raw token, so it read like a deleted key and lost its request and response

It now looks up only a saved value shaped like a virtual key's hash, reusing is_valid_sha256_hash, so OAuth2 batches follow their team's logging settings and the proxy-wide setting while deleted keys still fail closed

* fix(batches): hash the creator credential and write batch line items in parts

The creator credential reaches line metadata only as a sha256 hash, so an
OAuth2 token or the master key alias is never written to the bucket. A queued
store keeps only whether the batch was billed and downloads the output file
again at its turn, logging an error naming the batch if that fails. Lines are
built and uploaded one part at a time, each closing once its raw request and
result bytes reach BATCH_LINE_ITEMS_MAX_PART_BYTES (16 MiB by default), with
the input file indexed by byte span instead of parsed up front.

* docs(batches): say async_log_batch_line_items is called once per part

* fix(batches): log an error when a batch's deployment is gone before its line items are stored

---------

Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
2026-10-15 11:32:56 -07:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
base_sdk_tests test: extend runtime regression contracts for Messages and SDK (#46641) 2026-10-15 17:51:15 +00:00
benchmarks refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
code_coverage_tests fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) 2026-10-15 10:57:00 -07:00
documentation_tests fix(proxy): let configured pass-through endpoints outrank built-in provider prefix routes (#44133) 2026-10-10 12:26:17 -07:00
e2e test(ci): pin xai reported-cost split and coordinator-held Prisma lock timeout (#46645) 2026-10-15 04:08:56 -07:00
e2e_harness test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) 2026-10-12 13:54:03 -07:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
integration feat(batches): store each batch line's request and response in S3 and GCS (#41691) 2026-10-15 11:32:56 -07:00
rust-python-harness docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
test_litellm_rust test(rust-bridge): select native routes per test without LITELLM_RUST (#46620) 2026-10-15 09:12:08 -07:00
typing fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
unit feat(batches): store each batch line's request and response in S3 and GCS (#41691) 2026-10-15 11:32:56 -07:00
windows_tests test: move packaging and install checks off CircleCI build_and_test (#45757) 2026-10-10 09:33:51 -07:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.