* test: move guardrails, image, audio, batches, search legacy tests to unit
* test: restore the custom guardrail and remaining chunk 1 unit rows
* test: move the ocr matrix to e2e
* test: drop comments from moved tests and fix vertex ocr credentials
* test: move litellm utils and unified google legacy tests to their tiers
* test: move logging callback and pass-through unit legacy tests to their tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move store_model_in_db legacy tests and start the admin ui migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: finish the proxy job rows and drop their CircleCI jobs
Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI
* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names
* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist
* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in
* test: type the e2e beta header, prompt cache and secret manager values
* test: assert the missing model error message on the OpenAI routes
* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks
* test: make the langfuse fixture directory a package for the layout check
* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane
* test: keep the ocr e2e pdf fixture next to the test
* test: put meta below covers on the streaming tool search e2e
* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures
* test: count only OCR calls in the OCR matrix recorder
* test: pin OCR parse bodies and billed cost, group new e2e cases into classes
* ci: restore the python image anchor dropped in the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit
Relocate 53 legacy tests that pass with no network or keys into the tests/unit
files that mirror the code they exercise, drop 2 llm_guard tests already covered
by test_llm_guard_call_type_aliases, and delete the emptied legacy files.
* test: freeze the reset clock and run the real reset methods in the moved budget reset tests
Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await
the service-hook tasks the job spawns instead of sleeping. Drive the per-user
failure with a malformed budget_duration row from the prisma mock so the real
_reset_budget_for_key/user/team run instead of patched replacements.
* test: use fixed timestamps in the moved pass-through and spend log tests
* test: mock the moderation HTTP call and type the moved moderation and spend log tests
Serve the OpenAI moderation response through an httpx MockTransport so the real
async_make_request runs, annotate the moved tests' locals with Final, and pass
get_logging_payload typed kwargs instead of an untyped input_args dict.
* test: drive the moved proxy tests through boundaries and annotate their locals with Final
Serve GCS downloads and the Router moderation call through httpx mocks with fake
Google credentials, let the real pass-through success handler fail on the mocked
upstream response, and fold the Vertex live route endpoint check into the existing
route test. Annotate moved locals with Final, build the budget reset rows without
rebinding, and type the remaining fakes without **kwargs
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
litellm_llm_api_latency_metric, litellm_llm_api_time_to_first_token_metric,
litellm_request_total_latency_metric, and litellm_deployment_latency_per_output_token
previously carried requested_model/litellm_model_name/model_id but not model_group,
so pooled-deployment latency couldn't be grouped by model pool on dashboards --
only the proxy-overhead-only metrics (litellm_overhead_latency_metric and friends)
had model_group. All four metrics read enum_values.model_group through the existing
prometheus_label_factory plumbing, so no new parameter threading was needed, just
the label-list addition.
Co-authored-by: ahamedshaik16 <24526479+ahamedshaik16@users.noreply.github.com>
* refactor(types): prove runtime types at harness, search, rag and client boundaries
Replace Any with adapter-validated types in the litellm.agent() harness, the
search provider transformations, RAG ingestion and query, the vector store
pre-call hook and registry, the galileo and opik logging integrations and the
proxy client CLI. Each boundary gets unit tests for well-formed and malformed
payloads.
* chore(typing): prove types at more provider boundaries and restore search transformations
Second pass over a2a, embedding, rerank, image, audio and small provider
modules. Search transformations go back to their previous form because
validating their response bodies would change the proxy status for
malformed upstream bodies from 400 to 500.
* refactor(types): prove types at logging, files, rerank, image, audio and management boundaries
Replace Any with validated or annotation-only types in 37 more files: logging
integrations, token counters, provider files/rerank/image generation/audio
transcription transformations, pass-through logging handlers and management
endpoints. No proxy HTTP status or error type changes.
* refactor(types): prove types at repository, spend, files and router boundaries
Replace Any with repository table accessors, validated mappings and
annotation-only types in 33 more files: Prisma repositories, the enterprise
batch and responses cost checkers, budget reservation, files endpoints,
management endpoints, the policy registry, the adaptive and complexity routers
and the secret managers. No proxy HTTP status or error type changes.
* test(types): run the aiohttp transformation test in-process and cover repository row conversion
The aiohttp chat transformation test no longer starts a server. It feeds the
transformation a response whose json() returns the body under test.
The proxy unit shards now exercise stored model rows whose params are JSON
strings and the object permission create and update paths.
* feat(proxy): email alerts at configured percentages of a team member budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): label team member budget crossings as team member budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover the team member alert dispatch from _check_team_member_budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(email): drop the emoji from the team member budget alert template
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): ignore team member alert thresholds outside 1 to 100 on both the backend and the dashboard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound team member alert threshold key length before int parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the legacy covers marker from the team member alert test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team): reject malformed team_member_max_budget_alert_emails on team writes
Thresholds outside 1-100, non-list recipients, and invalid emails now return 422 on
/team/new, /team/update and PATCH /team/{id} instead of being stored and silently
ignored. The value is stored canonically. Read-side LiteLLM_TeamTable is unchanged,
and the PATCH body stays a raw merge patch so a null threshold still deletes it.
* fix(auth): enforce and alert on team member budgets only in common_checks
The builder re-checked the team member budget inline before common_checks ran the same
check, so one request that crossed a team_member_max_budget_alert_emails threshold
dispatched two alerts. Drop the inline check; common_checks is the single authorization
point and already covers per-member rows, the team default member budget, zero-cost
skips and the cross-pod spend counter. Its 422 message now uses the TeamMember=user:team
form the builder and budget reservation already returned.
* Revert "fix(team): reject malformed team_member_max_budget_alert_emails on team writes"
This reverts commit 703e754b46.
* fix(alerting): keep BaseBudgetAlertType.get_event_message zero-arg
Requiring user_info broke existing callers and out-of-tree subclasses. The team member
label now comes from SlackAlerting.budget_alerts, so the interface and its Readme are
unchanged from main.
* fix(mcp): keep team member budget enforcement on the MCP OAuth auth dependency
The MCP OAuth dependency stops at _user_api_key_auth_builder and never reaches common_checks, so removing the builder's inline member budget check would have let over-budget members through there. Enforce it explicitly for that caller.
* fix(auth): keep main's team member budget enforcement, alert once per request
Restore the builder's team member budget check and 422 message exactly as on main and drop the MCP-only gate. The builder sends the member alert only on the request it rejects; common_checks sends it for requests that get past the builder, so no request alerts twice.
* test(integration): read team member alert deliveries without a shared accumulator
* test(integration): match team member alert deliveries by subject so other alerts cannot race the count
* refactor(proxy): build the team member alert threshold config without mutable collections
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): collapse the alert recipient isinstance checks into one call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read the SMTP sink through lock-guarded snapshots and assert the exact deliveries
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_update_config_success_callback_normalization replaced
proxy_server.proxy_logging_obj with a MagicMock and never restored it.
Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping,
end-user and MCP tests on the same xdist worker awaited that mock and
failed. The test now uses monkeypatch.
test_prometheus_logging_callbacks set verbose_logger to DEBUG and
litellm.set_verbose at import, so every worker in the unit job ran with
DEBUG on. That broke caplog equality in the JEV classifier test, the
vertex streaming memory ratio, and four event-loop lag checks. The
module-level setup is removed; nothing in the file depended on it.
#43081 removed the OCR harness modules but left them in the
importability parametrize list.
test_get_model_info_bedrock_region reassigned litellm.model_cost and set
LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared
the get_model_info caches, so it failed whenever an earlier test had
looked up the regional model. It now uses monkeypatch and invalidates
the caches; the local_testing isolation fixture also invalidates them
after restoring model_cost.
The Windows job hit CircleCI's 10 minute no-output limit while cargo
compiles the Rust crates inside uv sync and uv build. Those two steps now
allow 30 minutes of silence.
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test(unit): make bedrock collector and secret scan timing tests deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): count interpreter calls instead of wall clock in the secret scan scaling test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): profile the secret scan with cProfile, restore the outer profiler and tighten the scaling bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>