mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
15 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
e319bf270c
|
feat(langfuse): migrate the sdk callback to langfuse v4 (#36741)
* feat(langfuse): migrate the sdk callback to langfuse v4
Replace the v2 trace()/generation()/span() calls with SDK v4 observations exported over OpenTelemetry, with one isolated tracer provider per Langfuse credential set, a discarding exporter for mock mode, and v4 trace and observation id normalization. Keeps the session-header trace provenance logic from main so each call under a session alias still gets its own trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): drop the always-true prompt client check now that v4 get_prompt is non-optional
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): isolate the e2e sync test from cached clients and log the real sdk major
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the slack trace-url lookup and drop dead v2 test shims
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(slack): cover the langfuse trace url built from the logger host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): pin langfuse to the locked 4.15.2 in the pip image
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): hash all-zero trace and observation ids instead of passing them through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(langfuse): honour caller generation ids and assert v4 OTLP exports in legacy tests
v2 accepted generation(id=...). v4 derives the observation id from the OTel
span id, so the isolated tracer provider now carries an id generator that
hands out the id start_generation asked for through a context variable, and
the callback passes the resolved generation_id metadata into it.
The legacy e2e suite patched httpx.Client.post and compared v2 ingestion
batches; it now patches requests.Session.post, decodes the OTLP protobuf
and compares the exported generation against regenerated fixtures. The
local readback test replaces the removed get_generations() with
api.observations.get_many() and polls Langfuse Cloud instead of sleeping.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): read the sdk version header from package metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): propagate trace_metadata as trace-level attributes in v4
v2 wrote trace(metadata=...) onto the trace object. In v4 the trace only
carries what the observations propagate, so a continuation request with
update_trace_keys=["trace_metadata"] updated the generation's metadata
while the trace kept its stale values. Coerce each entry to the SDK's
string limit and hand it to propagate_attributes(metadata=...).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): propagate interrupts raised during deferred client teardown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): honor ssl_verify=False and SSL_VERIFY on the v4 OTLP exporter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): fall back to the default CA when the configured bundle path is missing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): renew the client when eviction lands before the callback lease
The cache can evict a logger between handing it to the callback and the callback taking its
lease. Such a lease now hands back a fresh client acquired through the same parameters, so that
callback exports through a live tracer provider instead of one teardown already shut down.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): emit litellm_call_id and response_id as generation metadata
v2 put the provider response id inside the generation id. v4 observation ids are 16 hex chars derived from that string, so the ids move to generation metadata to keep generations searchable by response id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): read the response id through a typed protocol
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): do not claim trace root when continuing an existing trace
Langfuse derives a trace's name and I/O from any observation flagged
langfuse.internal.as_root, so a request carrying existing_trace_id
renamed the trace to the generation name and replaced the trace input
and output on every continuation. v2 only updated the keys listed in
update_trace_keys. Continuations now export as plain children of the
remote parent and keep the explicit langfuse.trace.* attributes for the
fields they do want changed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): iterate lease renewal instead of recursing, monkeypatch update_trace_keys flag in tests
The recursive lease fallback tripped tests/code_coverage_tests/recursive_detector.py; the renewal
candidates are now walked with itertools.chain. The six update_trace_keys tests set the litellm
global through pytest monkeypatch so the TQ008 budget stays within its ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): retry raised OTLP exports and honor LANGFUSE_TIMEOUT
The OTLP http exporter only retries 429 and 5xx; a connect or read timeout
propagates and BatchSpanProcessor drops the batch. Wrap the exporter in
RetryingSpanExporter (three backoff retries, as the v2 consumer did) and
build it on every path so the default and private-CA deployments share the
same channel, timeout and retry behaviour
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): sample on a hash of the full trace id and tolerate bad LANGFUSE_SAMPLE_RATE
TraceIdRatioBased reads the low 64 bits of the trace id. litellm trace ids are
UUIDs, whose variant bits sit at the top of that word, so every fractional rate
up to 0.5 dropped all traces. A SHA-256 of the full id gives an unbiased,
deterministic decision. Values outside [0, 1] or non numeric now warn and export
everything instead of raising during callback construction, which surfaced as a
500 on the first request of each worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): put the Langfuse trace link back into Slack alerts
The proxy registers LangfusePromptManagement for callbacks: ["langfuse"], so the alert helper never saw the literal "langfuse" string and returned before looking up the trace id, and the prompt management logger never stored the trace id it got back from log_event_on_langfuse. Recognize LangFuseLogger instances in the callback list, record the returned trace id in the shared service trace id cache, and skip the link when no trace id arrives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): relock langfuse 4.15.2 and opentelemetry 1.33.1 on current main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(langfuse): mark the deliberate blind except in client teardown for the strict ruff gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): pass the resource attributes mapping straight to Resource.create
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): warn about ignored UPSTREAM_LANGFUSE_* on the shared client init path too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): normalise the OTLP export path so a trailing host slash never yields a double slash
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): nest guardrail and grounding spans under the generation
Langfuse v4 derives the trace name and I/O from every observation marked as_root, and the one with the latest start time wins. Guardrail and grounding spans used to claim root next to the generation, so a post_call guardrail could replace the model's request and response on the trace with its own. Only the generation claims root now; the sibling spans become its children
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): rebuild the cached bundle when mock mode or sample rate changes
The SDK keys resource bundles on the public key alone, so a bundle built with the discarding exporter for LANGFUSE_MOCK, or with an earlier LANGFUSE_SAMPLE_RATE, was handed back to a client that asked for a live exporter or a different rate. Compare both when deciding whether the cached bundle is still valid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): keep trace_public true when a guardrail span is exported
Langfuse folds langfuse.trace.public across every observation in the trace and reads a missing attribute as false, so a guardrail child span without the flag turned a trace_public: true request private on Langfuse Cloud. Child spans now repeat the generation's value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): emit observations as plain OTel spans, keep the SDK for prompts and auth
The callback now owns an isolated TracerProvider and OTLP exporter and builds generation and child spans with public OpenTelemetry APIs plus the LangfuseOtelSpanAttributes constants. Caller trace ids, generation ids, parent observation ids and historical start and end times are honoured through the OTel id generator, remote SpanContext and explicit span timestamps, so no private Langfuse SDK tracing handle is used any more. The Langfuse client stays only for get_prompt and auth_check
This also resolves the gauntlet findings on the previous draft: fresh traces start from an empty context so caller application spans are never stamped, the Slack trace link is read from the request logging state instead of constructing a logger per alert, a truthy non-mapping trace_metadata is serialized instead of raising, trace_input and trace_output land on the root generation, discarding a cached client is done under the lock, and the prompt cache no longer leaks a task manager because the client cache no longer tears down shared providers
Fixtures under tests/logging_callback_tests lose the SDK-private langfuse.internal.as_root marker; every other exported attribute is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): hand the SDK client a validated sample rate so an unusable LANGFUSE_SAMPLE_RATE no longer breaks the callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): gate the SDK version before importing the OTel module in prompt management
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush every export channel on proxy shutdown and use the callback's host in Slack trace links
The shutdown hook imported litellm.utils.langFuseLogger, a global the callback registry never assigns, so a graceful restart dropped the spans still queued in the batch processors. Shutdown now calls flush_langfuse_tracing, which force-flushes every acquired channel. The Slack alert link falls back to the registered LangFuseLogger's langfuse_host when the request carries no dynamic host, and the export endpoint tests pin that scheme-relative or absolute LANGFUSE_OTEL_TRACES_EXPORT_PATH values stay on the configured host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): store resolved credentials on LangfusePromptManagement
The Slack alert trace link reads langfuse_host from every registered LangFuseLogger. Prompt management subclasses it without calling the parent constructor, so it never set the attribute and the alerting handler crashed before posting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush every export channel concurrently under one shutdown deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush export channels on daemon threads so a stuck channel cannot hold up interpreter exit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): own the tracer config and drop the SDK client for prompts and auth
The callback's TracerProvider now sets its sampler, span limits and id generator explicitly so unrelated OTEL_* variables no longer change what Langfuse receives, and trace metadata is written once on the trace instead of folded into the generation, which kept input and output under the attribute cap. Spans are emitted under the langfuse-sdk scope so Langfuse renders them natively, the batch processor queues 100k spans and honors LANGFUSE_FLUSH_AT, and the proxy shutdown flush runs off the event loop with a 10s deadline and logs a miss.
Prompts, auth_check and the project id now go through LangfuseAPI directly with a litellm-owned TTL cache, so no Langfuse() client is built and a host application's client on the same public key is left alone. Dead attributes, the unreachable exporter branch and the export list are cleaned up, and the client-budget eviction behavior is documented.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): export OTLP spans and fetch prompts through litellm's HTTPHandler instead of a private requests session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): gate the SDK version before importing the tracing module and retire unheld export channels
An installed v2 SDK used to fail inside the langfuse_sdk import and surface as "Langfuse not installed"; the version check now runs first so v2 users get the upgrade message, and only PackageNotFoundError means the package is missing
Export channels are now leased per credential set: acquire adds a holder, LangFuseLogger.stop (called by DynamicLoggingCache on expiry) releases one, and a channel with no holders is flushed and shut down after a 60 s grace, so rotating key or team credentials no longer grows one batch thread per credential set for the life of the process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): end the generation when a child span fails, take the client slot last, keep prompt cache keys structured
Generation spans now end in a finally block so a bad guardrail or provider entry cannot strand the trace. The logger acquires its export channel and REST client before counting a client slot and releases the channel synchronously if the REST client fails to build, so retries after a bad config do not exhaust the budget. LANGFUSE_TIMEOUT accepts decimals for the REST client like it already did for OTLP export. The prompt cache keys on (name, version, label) so a missing label and the literal label None stay apart
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): claim the cache entry before releasing its slot and channel hold on eviction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): coerce generation names, keep v2 release, timeout and retry defaults, refresh stale prompts off the loop
A non-string metadata generation_name reached the OTLP encoder and took the whole batch down; it is now exported as its text and the exporter drops only the span the encoder rejects. LANGFUSE_RELEASE falls back to the deploy platform's commit variable again, the export deadline is back to the v2 default of 20 s and LANGFUSE_MAX_RETRIES sizes the retry ladder. An expired prompt is served at once while one background thread refreshes it, a re-acquired export channel cancels the pending retire timer, and flush reports delivery rather than a drained queue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the current Langfuse shutdown flush warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): keep host OTel resource out, carry big metadata ints, tolerate bad flush and TTL env, stamp trace I/O under a parent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): name a malformed prompt cache TTL before the SDK import, keep metadata ints JSON safe, retry every 5xx export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): name the auth check failure, split a 413 export, wire LANGFUSE_DEBUG, stamp error output under a parent, send the ingestion version header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): honor LANGFUSE_DEBUG on the callbacks path, cap retry backoff, name the auth failure status and body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): cap LANGFUSE_MAX_RETRIES at 1000 so an absurd value cannot stall callback init
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): fold 413 halving into bounded rounds instead of recursion
The code-quality recursive-function gate flagged LangfuseSpanExporter.export. A batch of n spans settles within n.bit_length() halving rounds, so the split is a reduce over a frozen round state with the same posts, logs and results. The TTL gate test now asserts the gate returns without raising
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): truncate a single oversized span like v2 instead of dropping it, no retries on REST auth and project lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): write the metadata truncation marker under a flattened key so Langfuse keeps it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): patch the HTTPHandler export path and sync the metadata fixture and lease registry with the v4 callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): give the 413 split helpers a single explicit return path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): url-encode prompt names and fetch cold prompts without client retries
A cold get_prompt runs inline on the event loop; the generated v4 client's default two retries slept through
Retry-After (up to 60 s per attempt) and held the loop. The wrapper also passed the raw name into
api/public/v2/prompts/{name}, so 'what?' fetched prompt 'what' and folder names left the route. Quote the
name with safe='' like the v4 SDK's own get_prompt and pass max_retries=0 like the projects.get calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): retry a cold prompt miss once and drop upstream headers from prompt errors
A cold prompt fetch makes one immediate second attempt after a 5xx or a
transport failure, as the v2 client did, still with the generated client's
sleeping retries and Retry-After handling off so the event loop never stalls.
A failed fetch raises LangfusePromptError carrying only the status and body,
so the proxy no longer forwards Langfuse's response headers to its client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): stub the logger in the health auth_check test instead of dialing a closed port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): integration test for OTLP v4 delivery and prompt fetch through a real proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): keep the pip image's langfuse and otel pins on the v2 line its litellm 1.83.0 wheel expects
The image validates the published PyPI artifact, whose langfuse callback still
reads langfuse.version, so the 4.15.2 pin broke that callback. The pins move
together with the next LITELLM_VERSION bump. Also rewords the trace_version
precedence test docstring: v2 carried two version fields, v4 has one per span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
yuneng-jiang
|
4f93e2c3da
|
test: point CircleCI-only suites at models still in the cost map (#42617)
* test: point CircleCI-only suites at models still in the cost map #42435 removed cost map entries past their deprecation date and #42437 added litellm_uisettings to the config-synced tables, but both only updated tests/test_litellm. The CircleCI-only suites (local_testing, llm_translation, logging_callback_tests, litellm_utils_tests, unit) kept using the removed models or the old table list and went red on main. Each test keeps its assertions and swaps the removed model for a current one with the same provider and capabilities. The fireworks tests pick a vision model from the cost map because #34941 set supports_vision false on minimax-m3, and the vertex image provider test injects the image model set because #42435 removed every vertex_ai-image-models entry. * test(vertex_ai): register the image model through add_known_models in the provider test |
||
|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
David Chen
|
d1df4e838b
|
Litellm fix update bedrock models (#24947)
* update bedrock models in tests * updated more tests and model_prices_and_context_window * fix model id and pricing * replace more sonnet models * update tests * git push * update pricing * flaky total cost * monkey patch * relax the cost change * fix and revert some changes * revert the pricing * chore: move cost/pricing changes to bedrock-cost-fixes branch * chore: split Bedrock file-api beta stripping to separate branch Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch; see litellm_bedrock_invoke_strip_file_api_betas for that fix. Made-with: Cursor |
||
|
Ishaan Jaff
|
8e61b32b8e
|
[Staging] - Ishaan March 17th (#23903)
* feat(xai): add grok-4.20 beta 2 models with pricing (#23900)
Add three grok-4.20 beta 2 model variants from xAI:
- grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent)
- grok-4.20-beta-0309-reasoning (reasoning)
- grok-4.20-beta-0309-non-reasoning
Pricing (from https://docs.x.ai/docs/models):
- Input: $2.00/1M tokens ($0.20/1M cached)
- Output: $6.00/1M tokens
- Context: 2M tokens
All variants support vision, function calling, tool choice, and web search.
Closes LIT-2171
* docs: add Quick Install section for litellm --setup wizard (#23905)
* docs: add Quick Install section for litellm --setup wizard
* docs: clarify setup wizard is for local/beginner use
* feat(setup): interactive setup wizard + install.sh (#23644)
* feat(setup): add interactive setup wizard + install.sh
Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that
guides users through provider selection, API key entry, and proxy config
generation, then optionally starts the proxy immediately.
- litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu
(OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts,
port/master-key config, and litellm_config.yaml generation
- litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard
- scripts/install.sh: curl-installable script (detect OS/Python, pip
install litellm[proxy], launch wizard)
Usage:
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh
litellm --setup
* fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs
* fix(install.sh): install from git branch so --setup is available for QA
* fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error
* fix(install.sh): force-reinstall from git to bypass cached PyPI version
* fix(install.sh): show pip progress bar during install
* fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary
* fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists)
* fix(install.sh): suppress RuntimeWarning from module invocation
* fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing
* fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary
* fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe
* fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster
* feat(setup_wizard): arrow-key selector, updated model names
* fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm
* feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start
* style(install.sh): show git clone warning in blue
* refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils
* address greptile review: fix yaml escaping, port validation, display name collisions, tests
- setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys
- setup_wizard.py: add _styled_input() with readline ANSI ignore markers
- setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture
- setup_wizard.py: validate port range 1-65535, initialize before loop
- setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai
- setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict
- setup_wizard.py: skip model_list entries for providers with no credentials
- setup_wizard.py: prompt for azure deployment name
- setup_wizard.py: wrap os.execlp in try/except with friendly fallback
- setup_wizard.py: wrap config write in try/except OSError
- setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite)
- setup_wizard.py: add .gitignore tip next to key storage notice
- setup_wizard.py: fix run_setup_wizard() return type annotation to None
- scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh)
- scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch)
- scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat
- scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies
- tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape
* style: black format setup_wizard.py
* fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow
- guard termios/tty imports with try/except ImportError for Windows compat
- quote master_key as YAML double-quoted scalar (same as env vars)
- remove unused port param from _build_config signature
- _validate_and_report now returns the final key so re-entered creds are stored
- add test for master_key YAML quoting
* fix: add --port to suggested command, guard /dev/tty exec in install.sh
* fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change
* fix: address greptile review comments
- _yaml_escape: add control character escaping (\n, \r, \t)
- test: fix tautological assertion in test_build_config_azure_no_deployment_skipped
- test: add tests for control character escaping in _yaml_escape
* feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908)
* feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897)
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth
Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers
can trust a single signing authority instead of every upstream IdP.
Enable in config.yaml:
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss,
aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless
MCP_JWT_SIGNING_KEY env var is set.
Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration
so MCP servers can verify LiteLLM-issued tokens via OIDC discovery.
* Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message.
* fix: address P1 issues in MCPJWTSigner
- OpenAPI servers: warn + skip header injection instead of 500
- JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent
- sub claim: fallback to apikey:{token_hash} for anonymous callers
- ttl_seconds: validate > 0 at init time
* docs: add MCP zero trust auth guide with architecture diagram
* docs: add FastMCP JWT verification guide to zero trust doc
* fix: address remaining Greptile review issues (round 2)
- mcp_server_manager: warn when hook Authorization overwrites existing header
- __init__: remove _mcp_jwt_signer_instance from __all__ (private internal)
- discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation
- test docstring: reflect warn-and-continue behavior for OpenAPI servers
- test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs)
* fix: address Greptile round 3 feedback
- initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured
mode silently bypasses JWT injection, which is a zero-trust bypass
- _build_claims: remove duplicate inline 'import re' (module-level import already present)
- _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing
for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes
Addresses all missing pieces from the scoping doc review:
FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri
and token_introspection_endpoint. When set, the incoming Bearer token is
extracted from raw_headers (threaded through pre_call_tool_check), verified
against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if
valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode.
FR-12 (Configurable end-user identity mapping): end_user_claim_sources
ordered list drives sub resolution — sources: token:<claim>, litellm:user_id,
litellm:email, litellm:end_user_id, litellm:team_id.
FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always
override), remove_claims (delete) applied in that order.
FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a
second JWT injected as x-mcp-channel-token: Bearer <token>.
FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any
listed claim is absent; optional_claims passes listed claims from verified
token into the outbound JWT.
FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid,
sub, iss, exp, scope.
FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also
fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission).
P1 fixes:
- proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than
replaces extra_headers, preserving headers from prior guardrails.
- mcp_server_manager.py: warns when hook injects Authorization alongside a
server-configured authentication_token (previously silent).
- mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and
extracts incoming_bearer_token so FR-5 verification has the raw token.
- proxy/utils.py: remove stray inline import inspect inside loop (pre-existing
lint error, now cleaned up).
Tests: 43 passing (28 new tests covering all FR flags + P1 fixes).
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core)
Remaining files from the FR implementation:
mcp_jwt_signer.py — full rewrite with all new params:
FR-5: access_token_discovery_uri, token_introspection_endpoint,
verify_issuer, verify_audience + _verify_incoming_jwt(),
_introspect_opaque_token()
FR-12: end_user_claim_sources ordered resolution chain
FR-13: add_claims, set_claims, remove_claims
FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token
FR-15: required_claims (raises 403), optional_claims (passthrough)
FR-9: debug_headers → x-litellm-mcp-debug
FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list
mcp_server_manager.py:
- pre_call_tool_check gains raw_headers param to extract incoming_bearer_token
- Silent Authorization override warning fixed: now fires when server has
authentication_token AND hook injects Authorization
tests/test_mcp_jwt_signer.py:
28 new tests covering all FR flags + P1 fixes (43 total, all passing)
* fix(mcp_jwt_signer): address pre-landing review issues
- Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is
already populated and consumed by MCPJWTSigner in the same PR
- Fix _get_oidc_discovery to only cache the OIDC discovery doc when
jwks_uri is present; a malformed/empty doc now retries on the next
request instead of being permanently cached until proxy restart
- Add FR-5 test coverage for _fetch_jwks (cache hit/miss),
_get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt
(valid token, expired token), _introspect_opaque_token (active,
inactive, no endpoint), and the end-to-end 401 hook path — 53 tests
total, all passing
* docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features
Add scenario-driven sections for each new config area:
- Verify+re-sign with Okta/Azure AD (access_token_discovery_uri,
end_user_claim_sources, token_introspection_endpoint)
- Enforcing caller attributes with required_claims / optional_claims
- Adding metadata via add_claims / set_claims / remove_claims
- Two-token model for AWS Bedrock AgentCore Gateway
(channel_token_audience / channel_token_ttl)
- Controlling scopes with allowed_scopes
- Debugging JWT rejections with debug_headers
Update JWT claims table to reflect configurable sub (end_user_claim_sources)
* fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail
The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner.
All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri,
end_user_claim_sources, add/set/remove_claims, channel_token_audience,
required/optional_claims, debug_headers, allowed_scopes, etc.) were
silently dropped, making every advertised advanced feature non-functional
when loaded from config.yaml.
Add regression test that asserts every param is wired through correctly.
* docs(mcp_zero_trust): add hero image
* docs(mcp_zero_trust): apply Linear-style edits
- Lead with the problem (unsigned direct calls bypass access controls)
- Shorter statement section headers instead of question-form headers
- Move diagram/OIDC discovery block after the reader is bought in
- Add 'read further only if you need to' callout after basic setup
- Two-token section now opens from the user problem not product jargon
- Add concrete 403 error response example in required_claims section
- Debug section opens from the symptom (MCP server returning 401)
- Lowercase claims reference header for consistency
* fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL
- Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead.
Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks.
- Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h).
Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible.
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution
Fix 1: Run Black formatter on 35 files
Fix 2: Fix MyPy type errors:
- setup_wizard.py: add type annotation for 'selected' set variable
- user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment
Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total
spend instead of just 'any increase' from burst 2
Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979,
CVE-2026-27980, CVE-2026-29057
Fix 5: Fix router _pre_call_checks model variable being overwritten inside
loop, causing wrong model lookups on subsequent deployments. Use local
_deployment_model variable instead.
Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve
non-terminal-to-terminal path (matching the terminal path behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: regenerate poetry.lock to sync with pyproject.toml
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: format merged files from main and regenerate poetry.lock
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup)
Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in
test_router_region_pre_call_check, following the same pattern used in commit
|
||
|
Sameer Kankute
|
1a3d4c80db | Add JSON exact match test for vLLM embeddings | ||
|
Ishaan Jaffer
|
ee36c30217 | fix LF tests | ||
|
Alexsander Hamir
|
eaa04cd8ce
|
fix: use fastuuid helper (#14903)
* fix: use fastuuid helper across the codebase First batch of changes, simple drop in replacement. * second batch of changes * fixed: script mistake on helper file |
||
|
Krish Dholakia
|
066a502b89
|
Litellm dev 05 26 2025 p1 (#11165)
* fix(utils.py): prevent leaking sensitive keys to langfuse Fixes https://github.com/BerriAI/litellm/issues/11150 * test(langfuse/): unit test preventing future bedrock key leaks Fixes https://github.com/BerriAI/litellm/issues/11150 * test(test_langfuse_e2e_test.py): add unit test for vertex - make sure no key leaks occur * ci(test-litellm.yml): add pytest retry to github workflow * fix(proxy_server.py): support forwarding `/sso/key/generate` to the server root path url Fixes https://github.com/BerriAI/litellm/issues/10761 * fix(proxy_server.py): don't rewrite absolute path (PROXY_BASE_URL) with relative path (SERVER_ROOT_PATH) This causes issues when using a custom path with sso, when doing redirects * fix(utils.py): ignore token - will mistakenly redact 'max_tokens' as well |
||
|
Krish Dholakia
|
010a4d44af
|
Fix passing standard optional params (#11124)
* fix(main.py): use processed non-default-params as standard input params for langfuse Fixes https://github.com/BerriAI/litellm/issues/11072 Fixes https://github.com/BerriAI/litellm/issues/11096 * fix(main.py): rename variable to be more accurate * test(test_langfuse_e2e_test.py): add router unit test for langfuse e2e testing Prevent https://github.com/BerriAI/litellm/issues/11072 from happening again * build: update lock * fix(utils.py): refactor optional params function make it easier to get the standardized non default params * fix(utils.py): improve process non default params function * fix(main.py): include provider specific params in processed non default params used in logging ensures user can see any provider specific params on langfuse ensures user can see any provider specific params on langfus e |
||
|
Ishaan Jaff
|
b535c9bdc0
|
(Bug Fix - Langfuse) - fix for when model response has choices=[] (#8339)
* refactor _get_langfuse_input_output_content * test_langfuse_logging_completion_with_malformed_llm_response * fix _get_langfuse_input_output_content * fixes for langfuse linting * unit testing for get chat/text content for langfuse * fix _should_raise_content_policy_error |
||
|
Krish Dholakia
|
6834c5ecaf
|
Easier user onboarding via SSO (#8187)
* fix(ui_sso.py): use common `get_user_object` logic across jwt + ui sso auth Allows finding users by their email, and attaching the sso user id to the user if found * Improve Team Management flow on UI (#8204) * build(teams.tsx): refactor teams page to make it easier to add members to a team make a row in table clickable -> allows user to add users to team they intended * build(teams.tsx): make it clear user should click on team id to view team details simplifies team management by putting team details on separate page * build(team_info.tsx): separately show user id and user email make it easy for user to understand the information they're seeing * build(team_info.tsx): add back in 'add member' button * build(team_info.tsx): working team member update on team_info.tsx * build(team_info.tsx): enable team member delete on ui allow user to delete accidental adds * build(internal_user_endpoints.py): expose new endpoint for ui to allow filtering on user table allows proxy admin to quickly find user they're looking for * feat(team_endpoints.py): expose new team filter endpoint for ui allows proxy admin to easily find team they're looking for * feat(user_search_modal.tsx): allow admin to filter on users when adding new user to teams * test: mark flaky test * test: mark flaky test * fix(exception_mapping_utils.py): fix anthropic text route error * fix(ui_sso.py): handle situation when user not in db |
||
|
Ishaan Jaff
|
53a3ea3d06
|
(Refactor) Langfuse - remove prepare_metadata, langfuse python SDK now handles non-json serializable objects (#7925)
* test_langfuse_logging_completion_with_langfuse_metadata * fix litellm - remove prepare metadata * test_langfuse_logging_with_non_serializable_metadata * detailed e2e langfuse metadata tests * clean up langfuse logging * fix langfuse * remove unused imports * fix code qa checks * fix _prepare_metadata |
||
|
Ishaan Jaff
|
1f4ea88228
|
(Testing) - Add e2e testing for langfuse logging with tags (#7922)
* move langfuse tests * fix test * fix completion.json * working test * test completion with tags * langfuse testing fixes * faster logging testing * pytest-xdist in testing * fix langfuse testing flow * fix testing flow * fix config for logging tests * fix langfuse completion with tags stream * fix _verify_langfuse_call |