mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00

Mirror of https://github.com/BerriAI/litellm.git

Find a file

Mateo Wang bb448b0031 fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend (#28110 ) * fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend The image-edit cassettes for ``gpt-image-1`` were accumulating >50 episodes and being refused by the persister (``tests/_vcr_redis_persister.py``), so every CI run was hitting the real OpenAI endpoint. The async parametrize was the clearest tell: ``test_openai_image_edit_litellm_sdk[True]`` cached to 1 entry, but the ``[False]`` (async) sibling grew to 51 entries and never replayed. Two non-deterministic sources were fueling the growth, both fixed here. After this patch, the cassettes settle at one episode per unique call and replay for the 24-hour TTL like every other suite. 1. Pin httpx's multipart boundary at the source. The existing ``_normalize_multipart_boundary`` rewrites the boundary in the ``Content-Type`` header reliably, but on the async transport path the body is not always a contiguous ``bytes`` object when ``before_record_request`` runs, so the body-side replacement silently no-ops and the recorded cassette retains the random ``boundary=<hex>`` string. The next CI run gets a fresh random boundary, the ``safe_body`` matcher misses, and ``record_mode="new_episodes"`` appends another episode. Wrapping ``httpx._multipart.MultipartStream.__init__`` so it always uses ``vcr-static-boundary`` when no boundary is supplied eliminates the variance for both sync and async paths and leaves the normalizer in place as a backstop. Exposed as ``pin_httpx_multipart_boundary`` so other multipart-heavy suites (audio, ocr, batches) can adopt the same fixture later. 2. Pass raw ``bytes`` (not ``BytesIO`` streams) through the image-edit fixtures. A ``BytesIO`` whose file pointer is at EOF after the first multipart upload silently encodes an empty image on the next SDK / Router retry — yet another divergent body that VCR records as a new episode. ``bytes`` are immutable and position-less, so retries re-encode an identical payload every time. This is also a small production-correctness improvement: a customer passing ``BytesIO`` today would hit the same empty-body retry bug. The BytesIO-specific smoke test (``test_openai_image_edit_with_bytesio``) is preserved by giving ``get_test_images_as_bytesio`` its own factory instead of aliasing the bytes one. 3. Add ``scripts/flush_image_edit_vcr_cassettes.py`` — a one-shot Redis SCAN/DEL helper that clears the bloated pre-fix cassettes under ``litellm:vcr:cassette:tests/image_gen_tests/test_image_edits/``. Without this, the next CI run still loads the existing 51-entry cassette, the new fixed-boundary body still doesn't match any of the stale entries, the persister still refuses to save, and the bleed continues. Run once with the production ``CASSETTE_REDIS_URL`` after merge (dry-run by default). DIAGNOSTIC: log VCR body mismatches + per-episode body hashes Temporary observability boost so we can root-cause why ``test_image_edits.py`` async parametrizes still record fresh episodes on every CI run even though the multipart boundary is now pinned (sync parametrizes cache cleanly as VCR HIT). The matcher currently raises ``AssertionError("request bodies differ")`` with zero context, so we cannot tell whether the live body genuinely varies, the matcher is comparing a bytes object to a stream object, or the normalizer is silently skipping the body because it is not bytes/str. Three logs added; the first two are worth keeping permanently, the third is intended to be reverted after the diagnosis lands: 1. ``_safe_body_matcher`` now emits a structured stderr block on mismatch (type of each side, length, SHA-256, first divergent byte offset, ±100-byte window). Always-on -- mismatches are signal, not noise, and the existing per-test verdict already logs once per test. PERMANENT. 2. ``_normalize_multipart_boundary`` now logs to stderr when the body type is not bytes/bytearray/str -- the silent ``else: return`` branch was masking exactly the case we suspect is firing on async (httpx ``MultipartStream`` handed to vcrpy before the body is read). PERMANENT. 3. ``_RedisPersister.save_cassette`` now logs every episode's body SHA-256, length, and 120-byte preview at save time. This lets two consecutive CI runs be diffed: if the same test records a different hash run-to-run, the live body genuinely varies; if both runs record the same hash but the matcher still misses, the bug is in the matcher itself. TEMPORARY -- revert once the async variance is identified and fixed. Once a single ``image_gen_testing`` CI run produces these logs, revert this commit (or just the persister hash block) with a force push so the cassette save path is not noisy in steady-state. * DIAGNOSTIC: route VCR diagnostics through per-PID files (bypass xdist capture) Re-push of the diagnostic logging from the previous commit, this time wired so the output actually survives to the CI log. xdist captures stdout/stderr from every passing test in the worker process; the body-matcher and normalizer-skip diagnostics fire from inside vcrpy machinery during the test, so for any test that ultimately passes (which is all of them once the cassettes are recorded), the diagnostic lines are silently swallowed. Fix: write each diagnostic line to a per-PID file under ``test-results/vcr-diagnostics/<pid>.log`` instead of writing to stderr. The controller's ``pytest_terminal_summary`` aggregates those files and writes them through ``terminalreporter.write_line``, which is not subject to per-test capture. As a bonus, ``test-results/`` is already collected by the ``store_test_results`` step in CircleCI, so the raw per-worker logs survive as build artifacts even after the test session ends. Three call sites updated: 1. ``_emit_body_mismatch_diagnostic`` (matcher) -- writes the structured type/length/sha/window block via ``vcr_diag_write_line``. 2. ``_normalize_multipart_boundary`` -- logs the silent-skip path (body not bytes/bytearray/str) the same way. 3. ``_maybe_log_episode_body_hashes`` (persister) -- replaces the ``_log.warning`` calls (which the root-logger config also swallows in CI) with ``vcr_diag_write_line``. Image-gen conftest is the only suite wired to dump the aggregated log at session end. Other suites can opt in by adding ``emit_vcr_diagnostic_log(terminalreporter)`` to their own ``pytest_terminal_summary``. The diagnostic dir is cleared at the start of each session (controller-only) so a local rerun does not mix output from prior runs. Same revert plan as the previous diagnostic commit: keep the matcher + normalizer skip diagnostics permanently (they only fire on signal events), revert the persister body-hash dump once the async variance is identified. * fix(tests): coalesce iterable request bodies before matching/recording Root cause of the residual async image-edit cassette leak. The diagnostic run for ``ba3915d9`` printed: [vcr-safe-body-matcher] request body mismatch body[a]: type='list_iterator' length=unknown sha256=N/A body[b]: type='list_iterator' length=unknown sha256=N/A httpx's async transport hands vcrpy a ``request.body`` that is a ``list_iterator`` over multipart chunks rather than a contiguous ``bytes`` blob. Two consequences: 1. ``_safe_body_matcher`` compares the two iterator objects with ``==``, which is identity comparison for arbitrary iterators - semantically identical multipart bodies never compare equal, and ``record_mode="new_episodes"`` appends a new episode on every CI run until the cassette crosses ``MAX_EPISODES_PER_CASSETTE`` and the persister refuses to save (this is exactly what the OVERFLOW warning has been catching). 2. ``_normalize_multipart_boundary`` short-circuits its ``else: return`` branch because the body is neither bytes nor str, so any residual random boundary characters in the body bytes are never rewritten. Sync requests do not hit this code path: httpx's sync transport hands vcrpy a single ``bytes`` body, so ``==`` works and the boundary normalizer runs as intended. That is why ``test_openai_image_edit_litellm_sdk[True]`` records to ``entries=1`` and replays cleanly while ``[False]`` (async) kept growing by one episode per run. Fix: add ``_materialize_iterable_body`` which coalesces an iterable ``request.body`` into ``bytes`` in-place. Call it from two places: * The top of ``_before_record_request``, so the boundary normalizer and the cassette serializer both see bytes from then on. * The top of ``_safe_body_matcher``, as defense in depth in case a future vcrpy code path invokes the matcher without first going through ``_before_record_request``. The vcrpy ``Request`` is a wrapper used for matching and recording; the underlying httpx transport sends its own request body separately, so replacing the iterator on the vcrpy wrapper does not starve the live HTTP send. After this lands the async parametrizes should flip from ``[VCR MISS:RECORDED] entries=N+1`` to ``[VCR HIT] entries=N`` on the next CI run, matching the sync side and dropping the residual ~$3/day to $0. * fix(tests): handle bytes_iterator + never leave an exhausted body Follow-up to `8e08272b`. The previous attempt at coalescing iterable request bodies bailed out (``return`` without writing ``request.body``) whenever it could not classify the chunk type. That was the wrong failure mode for one critical case: vcrpy sometimes presents the body as ``iter(some_bytes)``, whose Python type is ``bytes_iterator`` and which yields ``int`` byte values (0-255), not byte chunks. The old code saw an ``int`` chunk, hit the ``else: return`` branch, and left ``request.body`` pointing at the now-exhausted iterator. The post-fix diagnostic run made this loud: [vcr-safe-body-matcher] request body mismatch body[a]: type='bytes_iterator' length=unknown sha256=N/A body[b]: type='bytes_iterator' length=unknown sha256=N/A Every async image-edit test then ballooned from entries=2 to entries=10 in that single CI run -- the exhausted iterator meant the live multipart upload went out as an empty body, OpenAI returned 400, the SDK + flaky retries fired, each retry got a fresh iterator that my hook exhausted again, and ``new_episodes`` recorded each failed attempt as a new cassette episode. This patch: * Recognizes ``bytes_iterator`` (chunks are ``int``) and reconstructs the buffer via ``bytes(chunks)``. * Keeps the existing ``list_iterator``-over-bytes-chunks handling via ``b"".join(...)``. * Always writes a bytes value back to ``request.body`` after consuming the iterator. If the chunk shape is unrecognized, ``request.body`` is set to ``b""`` rather than left as an exhausted iterator. That is wrong in the sense of "we lost the body" but right in the sense of "the failure mode is now visible (live API call sends empty body and fails fast) instead of invisible (corrupt cassette grows silently)". Combined with the matcher diagnostic, any future regression in this code path will surface in the CI log immediately. Local verification covers ``bytes_iterator``, ``list_iterator`` over bytes chunks, generator over bytes chunks, empty iterator, already-bytes (idempotent), identical-content iterator equality in the matcher (now matches), and differing-content iterator inequality (still raises). * fix(tests): clear vcrpy's sticky _was_iter flag so materialized bodies stay bytes Actual root cause of the async image-edit cassette leak. The previous diagnostic run produced this dead giveaway: [vcr-episode-body-hash] ... episode[0]: body type='bytes_iterator' is not bytes/bytearray/str -- cannot hash [vcr-safe-body-matcher] request body mismatch body[a]: type='bytes_iterator' length=unknown sha256=N/A body[b]: type='bytes_iterator' length=unknown sha256=N/A Both sides of the matcher were ``bytes_iterator`` after the materializer had supposedly converted them to bytes. That made no sense until I read vcrpy's ``Request`` class. vcrpy's ``Request`` keeps two private flags that are set in ``__init__`` from the original body's type and never cleared by the setter: def __init__(self, method, uri, body, headers): self._was_file = hasattr(body, "read") self._was_iter = _is_nonsequence_iterator(body) ... @property def body(self): if self._was_file: return BytesIO(self._body) if self._was_iter: return iter(self._body) return self._body @body.setter def body(self, value): if isinstance(value, str): value = value.encode("utf-8") self._body = value # <-- does NOT touch _was_iter / _was_file So when httpx's async transport hands vcrpy an iterator body, ``_was_iter`` becomes ``True`` and stays there forever. Even after ``_materialize_iterable_body`` writes plain bytes via ``request.body = out``, the next read of ``.body`` re-wraps the stored bytes in ``iter()`` -- producing a fresh ``bytes_iterator`` that compares unequal to any other ``bytes_iterator`` via object identity. The matcher missed every time, the cassette grew by one episode per run, and the persister saw the same iterator type when trying to hash the body for the diagnostic log. Fix: after writing the materialized bytes, also force ``_was_iter`` and ``_was_file`` to ``False``. vcrpy exposes no public API for this, so we touch the private flags directly -- acknowledged as a pragmatic test-only hack with a clear unit boundary (the only call site is ``_materialize_iterable_body``). Local repro reproduces the exact production setup: ``Request('POST', url, iter(b'multipart-content'), {})`` on two sides, runs the matcher, asserts HIT. Verified the matcher hits on identical content and still raises on differing content. Should be the last fix needed. Existing cassettes that contain oddly-shaped bodies (lists of int chunks, etc. from the previous ``_was_iter=True`` save path) still match because the materializer canonicalises both sides to bytes before comparison -- no fourth re-flush required. * revert(tests): drop the temp per-episode body-hash diagnostic Removed now that `1c51ad13` has confirmed the root cause (vcrpy's sticky ``_was_iter`` flag making the body getter re-wrap stored bytes in ``iter()`` on every access). The hash dump did its job -- the post-1c51ad13 image_gen_testing run shows all five async image-edit tests as ``[VCR HIT]`` with stable entry counts and zero billing errors -- and is too noisy to keep on by default (over 100 lines per session at steady state). Kept permanently: * ``_safe_body_matcher`` mismatch diagnostic in ``_vcr_conftest_common.py``. Only fires on a body mismatch, which is signal worth surfacing whenever it happens. * ``_normalize_multipart_boundary`` "skipped" log line. Same rationale -- only fires when the body shape is something the normalizer cannot rewrite in place. * The ``test-results/vcr-diagnostics/<pid>.log`` per-PID file plumbing (``vcr_diag_write_line`` / ``emit_vcr_diagnostic_log``). Useful for any future diagnostic that needs to bypass xdist stdout/stderr capture; cheap to keep. * chore(tests): delete unused flush script + wire VCR diagnostic dump everywhere * Remove ``scripts/flush_image_edit_vcr_cassettes.py``. It was a one-shot helper for the initial cassette flush; the iterator and ``_was_iter`` fixes mean no future flush should be required, and the script was never run anywhere (the actual flushes happened inside the CI conftest via the temp hacks that have since been reverted). * The matcher mismatch + normalizer skip diagnostics already write per-PID files for every suite that imports the shared VCR plumbing, but ``emit_vcr_diagnostic_log`` -- the controller-side dump that surfaces those files into the CI log at session end -- was only wired into ``image_gen_tests``. Add the one-line call to the 12 sibling conftests that already use VCR so the diagnostics surface in any suite's terminal output if a body matcher ever misses. No new output in steady state -- the dump is a no-op when no diagnostics were recorded that session. * chore(tests): trim non-essential comments per project comment policy Strips docstrings, inline comments, and block comments that this PR introduced where the code itself was already self-evident. Keeps the few lines that document non-obvious behaviour (raw-bytes-not-BytesIO rationale on the image fixtures, the per-PID-files-bypass-xdist note on the diagnostic directory). Touches only comments this PR added -- no pre-existing comment is removed. Net: -161 lines of comment/docstring across 3 files, no code behaviour change. * chore(tests): forward *kwargs in pin_httpx_multipart_boundary wrapper Defensive against future httpx MultipartStream.__init__ adding new optional kwargs. Without the forward, the wrapper would silently drop them. No behaviour change today. chore(tests): canonicalize VCR matchers and surface shouldn't-happen branches Bundles the "follow-up cleanup PR" into this one so it does not get lost. Four small changes: 1. Introduce ``_canonical_body(req) -> (bytes, pre_type)`` and route ``_safe_body_matcher`` through it. The matcher now operates on bytes by construction; the "compare two iterator objects via ``==`` and silently get object-identity semantics" failure mode (which cost us this entire PR to diagnose) is structurally impossible to reintroduce. ``pre_type`` is the body type before canonicalization, surfaced by the mismatch diagnostic so a future regression involving a new body shape is still visible. 2. Add a structured diagnostic to ``_key_fingerprint_matcher``. It was previously raising a bare ``AssertionError("API key fingerprints differ")`` with zero context -- exactly the anti-pattern the body matcher had before this PR. 3. Surface "shouldn't-happen" branches via ``vcr_diag_write_line``: * ``_strip_image_b64_payloads`` -- logs when ``response``, ``response['body']``, or ``response['body']['string']`` arrives in an unexpected shape (vcrpy contract violation). * ``_compute_key_fingerprint`` -- logs the ``"no-key"`` fallback with the request method/URL so a stripped-auth-header bug is visible instead of masked. * ``_canonical_body`` -- logs its own empty-bytes fallback when a body has a shape ``_materialize_iterable_body`` did not handle. 4. Re-introduce per-episode body-hash logging in ``_RedisPersister.save_cassette`` (was reverted in `927c5548` as "noisy"). Quantified cost: ~25 KB of CI log per session at peak, ~ms-scale CPU, zero output in steady state (no save = no log). Trade-off favours keeping it: lets two consecutive CI runs be diffed by body hash, which is how we will spot the next regression in the same class. All call sites still work: local repro confirms iter==iter HIT, iter!=iter raises, plain-bytes HIT, body-hash log emits via the same per-PID file plumbing as the matcher diagnostics. * chore(tests): symmetrize diag-log cleanup across every VCR-using conftest ``image_gen_tests/conftest.py`` was the only suite that cleared ``test-results/vcr-diagnostics/.log`` at session start. The other 12 VCR-using conftests inherited any stale per-PID logs from a previous local run and would dump them in the terminal summary -- harmless in CI (fresh container) but confusing locally when running multiple suites in sequence. Extracts the cleanup into a ``reset_vcr_diag_dir`` helper in ``tests/_vcr_conftest_common.py`` and calls it from every VCR-using conftest's ``pytest_configure``. Same single source of truth, no inline duplication. fix(tests): gate body materialization on __next__ and strip PR comments aiohttp/vcrpy stores the json kwarg as a dict; _materialize_iterable_body was iterating it via __iter__ and joining the keys, replacing the request body with concatenated key names ("textlanguageentities"). Gate on __next__ so containers (dict/list/tuple) are left alone — only single-use iterators like httpx's bytes_iterator / list_iterator are materialized. Log diagnostic line when chunk type is unrecognized. * fix(tests): JSON-encode dict bodies in canonical_body for stable matching aiohttp stubs store the json kwarg as a dict; the fallback that compared all dicts as b"" caused concurrent presidio analyze calls to be served the wrong cassette episode. JSON-encode with sort_keys for stable bytes. * fix(tests): guard emit_vcr_diagnostic_log against multi-conftest re-emission Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(tests): globalize multipart-boundary pin + stabilize whisper fixtures Diagnostic shows audio_testing was silently re-recording 50+ live Whisper episodes per CI run (over MAX_EPISODES_PER_CASSETTE, so the persister refused to save). Two changes: * Move the session-autouse _pin_multipart_boundary fixture into the shared _vcr_conftest_common module so every VCR-using suite picks it up via a single import. image_gen had it inline; the other 12 suites silently lacked it. * Replace the module-level open("rb") audio file handles in test_whisper with cached bytes + a per-call (filename, bytes, mimetype) tuple, mirroring the image_edits raw-bytes pattern. Stops the file-pointer- at-EOF bug where the second test got an empty multipart body. * chore(tests): drop per-episode body-hash dump and redundant emit guard --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>		2026-05-18 09:15:39 -07:00
.circleci	fix(ci): flag codecov uploads, enable carryforward, close coverage gaps (#28028 )	2026-05-16 10:56:32 -07:00
.devcontainer	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
.github	fix(ci): flag codecov uploads, enable carryforward, close coverage gaps (#28028 )	2026-05-16 10:56:32 -07:00
.semgrep/rules	security: remove .claude/settings.json and add semgrep rule to prevent re-adding	2026-03-25 11:57:43 -07:00
backend	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
ci_cd	Drop dep bumps + black-26 reformat to clear fork CI policy	2026-05-07 23:04:52 +00:00
cookbook	Drop dep bumps + black-26 reformat to clear fork CI policy	2026-05-07 23:04:52 +00:00
db_scripts	Drop dep bumps + black-26 reformat to clear fork CI policy	2026-05-07 23:04:52 +00:00
deploy	chore: remove legacy deployment artifacts and litellm-js packages (#27541 )	2026-05-09 20:51:34 +00:00
dist	build: update dependencies	2025-11-01 12:58:39 -07:00
docker	chore: remove legacy deployment artifacts and litellm-js packages (#27541 )	2026-05-09 20:51:34 +00:00
docs	fix(hosted_vllm): normalize custom tools for chat completions (#25763 )	2026-05-05 17:27:02 -07:00
enterprise	[Infra] Bump versions (#28094 )	2026-05-16 18:31:43 -07:00
gateway	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
helm/litellm	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
litellm	fix(proxy): gate team allowed_passthrough_routes to proxy admins (#28097 )	2026-05-18 08:56:25 -07:00
litellm-proxy-extras	bump: version 0.4.71 → 0.4.72	2026-05-13 21:51:11 -07:00
migrations	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
scripts	perf: eliminate per-request callback scanning on proxy hot path (#27858 )	2026-05-14 09:28:31 -07:00
terraform/litellm	feat: add Terraform stacks for deploying LiteLLM on AWS and GCP (#27673 )	2026-05-16 17:26:20 -07:00
tests	fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend (#28110 )	2026-05-18 09:15:39 -07:00
ui	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
.dockerignore	fix critical CVE vulnerabliltes (#20683 )	2026-02-07 22:23:01 -08:00
.env.example	Add new model provider Novita AI (#7582 ) (#9527 )	2025-05-12 21:49:30 -07:00
.flake8	chore: list all ignored flake8 rules explicit	2023-12-23 09:07:59 +01:00
.git-blame-ignore-revs	Add my commit to .git-blame-ignore-revs	2024-05-12 10:21:10 -07:00
.gitattributes	ignore ipynbs	2023-08-31 16:58:54 -07:00
.gitguardian.yaml	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
.gitignore	feat: add Terraform stacks for deploying LiteLLM on AWS and GCP (#27673 )	2026-05-16 17:26:20 -07:00
.npmrc	[Fix] CI/Tooling: Correct min-release-age value in .npmrc files	2026-04-29 19:49:27 -07:00
AGENTS.md	chore: remove legacy deployment artifacts and litellm-js packages (#27541 )	2026-05-09 20:51:34 +00:00
ARCHITECTURE.md	[Docs] Litellm architecture fixes 2 (#19252 )	2026-01-16 14:52:16 -08:00
CLAUDE.md	chore(mcp): warn on internal + upstream PKCE delegate	2026-05-15 10:05:35 +05:30
codecov.yaml	fix(ci): flag codecov uploads, enable carryforward, close coverage gaps (#28028 )	2026-05-16 10:56:32 -07:00
CONTRIBUTING.md	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
cosign.pub	[Infra] Add release workflow and cosign public key	2026-03-31 14:30:27 -07:00
docker-compose.hardened.yml	[Feature] Download Prisma binaries at build time instead of at runtime for Security Restricted environments (#17695 )	2025-12-16 21:25:53 +05:30
docker-compose.yml	feat: add read-replica routing for Prisma DB via DATABASE_URL_READ_REPLICA (#27493 )	2026-05-08 21:05:50 -07:00
Dockerfile	fix(docker): restore npm@11.14.0 lost in merge resolution	2026-05-07 17:25:10 -07:00
GEMINI.md	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
LICENSE	refactor: creating enterprise folder	2024-02-15 12:54:13 -08:00
license_cache.json	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
Makefile	tests(vcr): trim non-load-bearing comments and docstrings	2026-04-30 21:48:48 +00:00
mcp_servers.json	Add ScrapeGraph MCP server configuration (#18923 )	2026-01-11 21:57:46 +05:30
model_prices_and_context_window.json	feat(model_catalog): add Azure AI Foundry GPT-5.4 model metadata (#28030 )	2026-05-16 15:08:10 -07:00
package-lock.json	chore(deps): refresh dependency locks	2026-05-04 11:36:18 -07:00
package.json	chore(deps): refresh dependency locks	2026-05-04 11:36:18 -07:00
policy_templates.json	feat: Add Canadian PII protection (PIPEDA) (#22951 )	2026-03-06 18:27:31 -08:00
prometheus.yml	build(docker-compose.yml): add prometheus scraper to docker compose	2024-07-24 10:09:23 -07:00
provider_endpoints_support.json	feat(audio_transcription): add NVIDIA Riva STT provider (#27185 )	2026-05-05 17:17:51 -07:00
proxy_server_config.yaml	chore(ci): modernize model references in tests and configs (#27856 )	2026-05-15 15:44:28 -07:00
pyproject.toml	[Infra] Bump versions (#28094 )	2026-05-16 18:31:43 -07:00
pyrightconfig.json	Agents - support agent registration + discovery (A2A spec) (#16615 )	2025-11-14 18:23:30 -08:00
README.md	Merge pull request #26521 from BerriAI/litellm_docs_tweaks	2026-04-25 17:50:29 -03:00
render.yaml	build(render.yaml): fix health check route	2024-05-24 09:45:28 -07:00
ruff.toml	[Fix] CI: fix 6 more CircleCI job failures from uv migration	2026-04-10 21:06:25 -07:00
schema.prisma	feat(mcp): add delegate_auth_to_upstream flag for PKCE passthrough (#27834 )	2026-05-13 12:06:13 -07:00
security.md	chore: update security.md (#24871 )	2026-03-31 13:13:18 -07:00
taplo.toml	fix(agentcore): simplify agentcore streaming (#17141 )	2026-01-19 05:20:24 -08:00
uv.lock	[Infra] Bump versions (#28094 )	2026-05-16 18:31:43 -07:00

README.md

🚅 LiteLLM

LiteLLM AI Gateway

Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website

What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers

Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
Drop-in OpenAI compatibility — swap providers without rewriting your code
Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters

Netflix

Features

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

uv add litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

Agents - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway

Step 2. Call Agent via A2A SDK

from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key

async with httpx.AsyncClient(headers=headers) as httpx_client:
    resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)

    request = SendMessageRequest(
        id=str(uuid4()),
        params=MessageSendParams(
            message={
                "role": "user",
                "parts": [{"kind": "text", "text": "Hello!"}],
                "messageId": uuid4().hex,
            }
        )
    )
    response = await client.send_message(request)

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

Supported Providers (Website Supported Models | Docs)

Provider	`/chat/completions`	`/messages`	`/responses`	`/embeddings`	`/image/generations`	`/audio/transcriptions`	`/audio/speech`	`/moderations`	`/batches`	`/rerank`
Abliteration (`abliteration`)	✅
AI/ML API (`aiml`)	✅	✅	✅	✅	✅
AI21 (`ai21`)	✅	✅	✅
AI21 Chat (`ai21_chat`)	✅	✅	✅
Aleph Alpha	✅	✅	✅
Amazon Nova	✅	✅	✅
Anthropic (`anthropic`)	✅	✅	✅						✅
Anthropic Text (`anthropic_text`)	✅	✅	✅						✅
Anyscale	✅	✅	✅
AssemblyAI (`assemblyai`)	✅	✅	✅			✅
Auto Router (`auto_router`)	✅	✅	✅
AWS - Bedrock (`bedrock`)	✅	✅	✅	✅						✅
AWS - Sagemaker (`sagemaker`)	✅	✅	✅	✅
Azure (`azure`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
Azure AI (`azure_ai`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
Azure Text (`azure_text`)	✅	✅	✅			✅	✅	✅	✅
Baseten (`baseten`)	✅	✅	✅
Bytez (`bytez`)	✅	✅	✅
Cerebras (`cerebras`)	✅	✅	✅
Clarifai (`clarifai`)	✅	✅	✅
Cloudflare AI Workers (`cloudflare`)	✅	✅	✅
Codestral (`codestral`)	✅	✅	✅
Cohere (`cohere`)	✅	✅	✅	✅						✅
Cohere Chat (`cohere_chat`)	✅	✅	✅
CometAPI (`cometapi`)	✅	✅	✅	✅
CompactifAI (`compactifai`)	✅	✅	✅
Custom (`custom`)	✅	✅	✅
Custom OpenAI (`custom_openai`)	✅	✅	✅			✅	✅	✅	✅
Dashscope (`dashscope`)	✅	✅	✅
Databricks (`databricks`)	✅	✅	✅
DataRobot (`datarobot`)	✅	✅	✅
Deepgram (`deepgram`)	✅	✅	✅			✅
DeepInfra (`deepinfra`)	✅	✅	✅
Deepseek (`deepseek`)	✅	✅	✅
ElevenLabs (`elevenlabs`)	✅	✅	✅			✅	✅
Empower (`empower`)	✅	✅	✅
Fal AI (`fal_ai`)	✅	✅	✅		✅
Featherless AI (`featherless_ai`)	✅	✅	✅
Fireworks AI (`fireworks_ai`)	✅	✅	✅
FriendliAI (`friendliai`)	✅	✅	✅
Galadriel (`galadriel`)	✅	✅	✅
GitHub Copilot (`github_copilot`)	✅	✅	✅	✅
GitHub Models (`github`)	✅	✅	✅
Google - PaLM	✅	✅	✅
Google - Vertex AI (`vertex_ai`)	✅	✅	✅	✅	✅
Google AI Studio - Gemini (`gemini`)	✅	✅	✅
GradientAI (`gradient_ai`)	✅	✅	✅
Groq AI (`groq`)	✅	✅	✅
Heroku (`heroku`)	✅	✅	✅
Hosted VLLM (`hosted_vllm`)	✅	✅	✅
Huggingface (`huggingface`)	✅	✅	✅	✅						✅
Hyperbolic (`hyperbolic`)	✅	✅	✅
IBM - Watsonx.ai (`watsonx`)	✅	✅	✅	✅
Infinity (`infinity`)				✅
Jina AI (`jina_ai`)				✅
Lambda AI (`lambda_ai`)	✅	✅	✅
Lemonade (`lemonade`)	✅	✅	✅
LiteLLM Proxy (`litellm_proxy`)	✅	✅	✅	✅	✅
Llamafile (`llamafile`)	✅	✅	✅
LM Studio (`lm_studio`)	✅	✅	✅
Maritalk (`maritalk`)	✅	✅	✅
Meta - Llama API (`meta_llama`)	✅	✅	✅
Mistral AI API (`mistral`)	✅	✅	✅	✅
Moonshot (`moonshot`)	✅	✅	✅
Morph (`morph`)	✅	✅	✅
Nebius AI Studio (`nebius`)	✅	✅	✅	✅
NLP Cloud (`nlp_cloud`)	✅	✅	✅
Novita AI (`novita`)	✅	✅	✅
Nscale (`nscale`)	✅	✅	✅
Nvidia NIM (`nvidia_nim`)	✅	✅	✅
OCI (`oci`)	✅	✅	✅
Ollama (`ollama`)	✅	✅	✅	✅
Ollama Chat (`ollama_chat`)	✅	✅	✅
Oobabooga (`oobabooga`)	✅	✅	✅			✅	✅	✅	✅
OpenAI (`openai`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
OpenAI-like (`openai_like`)				✅
OpenRouter (`openrouter`)	✅	✅	✅
OVHCloud AI Endpoints (`ovhcloud`)	✅	✅	✅
Perplexity AI (`perplexity`)	✅	✅	✅
Petals (`petals`)	✅	✅	✅
Predibase (`predibase`)	✅	✅	✅
Recraft (`recraft`)					✅
Replicate (`replicate`)	✅	✅	✅
Sagemaker Chat (`sagemaker_chat`)	✅	✅	✅
Sambanova (`sambanova`)	✅	✅	✅
Snowflake (`snowflake`)	✅	✅	✅
Text Completion Codestral (`text-completion-codestral`)	✅	✅	✅
Text Completion OpenAI (`text-completion-openai`)	✅	✅	✅			✅	✅	✅	✅
Together AI (`together_ai`)	✅	✅	✅
Topaz (`topaz`)	✅	✅	✅
Triton (`triton`)	✅	✅	✅
V0 (`v0`)	✅	✅	✅
Vercel AI Gateway (`vercel_ai_gateway`)	✅	✅	✅
VLLM (`vllm`)	✅	✅	✅
Volcengine (`volcengine`)	✅	✅	✅
Voyage AI (`voyage`)				✅
WandB Inference (`wandb`)	✅	✅	✅
Watsonx Text (`watsonx_text`)	✅	✅	✅
xAI (`xai`)	✅	✅	✅
Xinference (`xinference`)				✅

Read the Docs

Get Started

You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:

	LiteLLM AI Gateway	LiteLLM Python SDK
Use Case	Central service (LLM Gateway) to access multiple LLMs	Use LiteLLM directly in your Python code
Who Uses It?	Gen AI Enablement / ML Platform Teams	Developers building LLM projects
Key Features	Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management	Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)

Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

Run in Developer Mode

Services

Setup .env file in root
Run dependant services docker-compose up db prometheus

Backend

(In root) create virtual environment python -m venv .venv
Activate virtual environment source .venv/bin/activate
Install dependencies uv sync --all-extras --group proxy-dev
uv run prisma generate
prisma generate
Start proxy backend python litellm/proxy/proxy_cli.py

Frontend

Navigate to ui/litellm-dashboard
Install dependencies npm install
Run npm run dev to start the dashboard

Verify Docker Image Signatures

All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Enterprise

For companies that need better security, user management and professional support

Get an Enterprise License Talk to founders

This covers:

✅ Features under the LiteLLM Commercial License:
✅ Feature Prioritization
✅ Custom Integrations
✅ Professional Support - Dedicated discord + slack
✅ Custom SLAs
✅ Secure access with Single Sign-On

Contributing

We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.

Quick Start for Contributors

This requires uv to be installed.

git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only

For detailed contributing guidelines, see CONTRIBUTING.md.

📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.

Code Quality / Linting

LiteLLM follows the Google Python Style Guide.

Our automated checks include:

Black for code formatting
Ruff for linting and code quality
MyPy for type checking
Circular import detection
Import safety checks

All these checks must pass before your PR can be merged.

Support / talk with founders

Schedule Demo 👋
Community Discord 💭
Community Slack 💭
Our emails ✉️ ishaan@berri.ai / krrish@berri.ai