mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
18 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
8a9d5b15b4
|
feat(langfuse): support langfuse_environment as a per-key dynamic callback param (#38264)
* feat(langfuse): support langfuse_environment as a per-key dynamic callback param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the langfuse_environment constructor param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): only pass environment when the SDK client supports it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): drop the request-body metadata test for langfuse_environment
The proxy bans request-body callback params by default (derived from
_supported_callback_params in auth_utils), so the metadata channel this
test asserted is rejected with a 401 on the proxy. The supported channel
is admin-set key/team callback_vars, with LANGFUSE_TRACING_ENVIRONMENT
as the deployment-wide fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(langfuse): validate langfuse_environment, avoid redundant clients, honor it in langfuse_otel
Closes the review gaps on the langfuse_environment param:
- Validate values against Langfuse's environment pattern at save time
(/key/generate, /key/update, /team callback all 400 on e.g. 'Production'
instead of 200-then-silently-dropping every trace server-side) and at
logger init; non-string values are str()-coerced instead of crashing
the SDK's regex check per event.
- Treat empty/whitespace values and values equal to the deployment-wide
LANGFUSE_TRACING_ENVIRONMENT as non-dynamic so an environment-only
override that changes nothing no longer mints a duplicate SDK client
against MAX_LANGFUSE_INITIALIZED_CLIENTS.
- langfuse_otel now reads the per-key/team langfuse_environment from
standard_callback_dynamic_params instead of only the env var.
- Advertise the param on the discovery surfaces: callback_configs.json
(langfuse + langfuse_otel), the dashboard callback registry, and the
/team/{team_id}/callback docstring (schema.d.ts regenerated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format langfuse files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lint): remove duplicate test import, LIT002 dict literal, and mock-echo otel test
- drop redundant in-function import of callback_config_error (F811)
- avoid the `or {}` mutable literal in _set_langfuse_specific_attributes (LIT002)
- rewrite the dynamic-env otel test to observe span.set_attribute output
instead of patching litellm internals (TQ002/TQ008)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
yucheng-berri
|
ba8d8b6e14
|
fix(logging): redact tool call arguments to valid JSON and preserve null content (#38182)
* fix(logging): redact tool call arguments to valid JSON and preserve null content Resolves LIT-6102 * refactor(logging): centralize redacted tool-call arguments constant and satisfy test-quality gate * fix(responses): drop Final annotations on loop-assigned locals flagged by basedpyright * fix(responses): skip custom tool calls in redacted-arguments normalizer * fix(logging): keep the redaction sentinel in stored tool-call arguments and preserve null output text |
||
|
yucheng-berri
|
55ec491d03
|
fix(otel): bound and shut down credential-scoped tracer providers (#36591)
* fix(otel): bound and shut down credential-scoped tracer providers Each credential-scoped TracerProvider owns a BatchSpanProcessor worker thread that only stops on shutdown, and the v1 cache holding them was an unbounded, unsynchronized dict that never shut anything down. Every distinct team/key credential set therefore added a thread for the life of the process, and concurrent first-requests for the same credential set orphaned duplicate providers outright. Make the cache a lock-guarded bounded LRU that shuts down whatever it drops, matching the v2 TenantTracerCache. Providers wrapping a caller-supplied SpanExporter instance share that exporter with the logger's own provider, so they are dropped without shutdown; those use SimpleSpanProcessor and own no thread. * fix(otel): reclaim dropped providers on a dedicated executor Sustained credential churn queues one blocking shutdown per eviction, so using the shared logging executor let an unreachable tenant endpoint stall unrelated logging work behind the OTLP retry budget. Give provider shutdown its own bounded pool; its threads spawn lazily, so a proxy that never evicts still pays nothing. * fix(otel): decide provider shutdown from the victim, not the evicting request Both dynamic entry points share one provider cache, so it can hold providers of mixed exporter ownership. Reading the ownership flag from the evicting request therefore stopped a shared caller-supplied exporter in one direction, silencing telemetry process-wide, and leaked a BatchSpanProcessor thread in the other. Cache ownership alongside the provider so the drop decision reads the victim's own flag. * fix(otel): honor the widened header mapping type instead of dict only Widening the header parameter to Mapping left the isinstance check on dict, so a non-dict Mapping silently returned no headers at all, which for the OTLP path means an unauthenticated exporter and no traces with nothing raised. The dict branch also returned the caller's own object, and dropping the defensive copy at the call site let that alias reach a long-lived exporter. Match on Mapping and copy. * fix(otel): do not give a provider we may never stop an interpreter-exit hook Every TracerProvider registers an atexit hook by default, and that hook holds a strong reference. Providers wrapping a caller-supplied exporter are dropped without shutdown, so they stayed pinned for the life of the process and then stopped the shared exporter at exit. Tie shutdown_on_exit to ownership: those providers use SimpleSpanProcessor and buffer nothing, so they lose no flush, while providers that own their exporter keep the hook and their exit flush. Also stop the victim the eviction test leaves behind, and trim the added comments. |
||
|
yucheng-berri
|
d86336a7c6
|
fix(langfuse): emit otel trace version and release on the keys langfuse v4 reads (#36702)
* fix(langfuse): emit otel trace version and release on the keys langfuse v4 reads The langfuse_otel exporter wrote version to langfuse.generation.version and langfuse.trace.version, and release to langfuse.trace.release. Langfuse v4 recognizes neither, so both landed in the generic span attribute bag and every trace reported version and release as null. v4 has a single langfuse.version key, lifted to the trace when it sits on the root span, plus langfuse.release. Also routes the otel v2 preset's per-request headers through the shared builder so key-scoped and team-scoped exports carry x-langfuse-ingestion-version like the other three exporter paths already do. * fix(langfuse): give trace_version precedence over version on the shared v4 key Matches the documented contract in docs/observability/langfuse_integration.md and the legacy langfuse SDK callback, which both treat trace_version as the authoritative trace version with version as its fallback. |
||
|
yucheng-berri
|
bd44c9e305
|
fix(langfuse): send v4 ingestion header for otel callback (#33907)
* fix(langfuse): send v4 ingestion header for otel callback * refactor(langfuse): inline otel ingestion header literals * test(langfuse): assert v4 ingestion header on dynamic key config paths * style: apply ruff format to langfuse otel header changes * chore(langfuse): drop stale development annotation on json import --------- Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com> |
||
|
Yassin Kortam
|
2162da5015
|
fix(langfuse_otel): build per-request OTLP exporter from key and team dynamic Langfuse credentials (#32437)
* fix(langfuse_otel): build per-request OTLP exporter from key/team dynamic Langfuse credentials Key-scoped langfuse_otel callbacks only injected Authorization headers into the init-time exporter, so a proxy without global LANGFUSE_* env vars kept its fallback exporter and never exported traces to Langfuse. Dynamic params now build a full per-request OTLP config (endpoint from the key's langfuse_host, otlp_http, basic auth from the key's credentials). Resolves LIT-3976 * fix(otel): log dynamic config endpoint in span processor debug output * fix(otel): redact authorization headers in exporter debug logs |
||
|
Sameer Kankute
|
079c136742
|
chore(oss): litellm oss staging 120626 (#30292)
* feat(bedrock): add bedrock mantle gemma 4 models (#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from #25776 Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on #30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. #27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes #30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes #27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes #27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes #27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes #27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes #29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR #29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes #28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request #29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to #30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Julio Quinteros Pro
|
81827be215 |
fix: prevent sys.modules["langfuse"] import failures in langfuse unit tests
Three test failures caused by the real langfuse SDK import being triggered
at test time:
1. test_langfuse_prompt_management.py: Both tests create LangfusePromptManagement()
which calls `import langfuse`. Since earlier TestLangfuseUsageDetails tests
remove sys.modules["langfuse"] via patch.dict teardown, the real langfuse
import runs and fails on Python 3.14 (pydantic v1 incompatibility).
Fix: add setup_method/teardown_method to mock sys.modules["langfuse"].
2. test_langfuse.py::test_max_langfuse_clients_limit: Same root cause — creates
LangFuseLogger() without mocking sys.modules["langfuse"].
Fix: wrap test body with patch.dict("sys.modules", {"langfuse": mock}).
3. test_langfuse_otel.py::test_extract_langfuse_metadata_with_header_enrichment:
Replaces sys.modules["litellm.integrations.langfuse.langfuse"] with a stub
without restoring it, causing patch() in later tests to target the stub
instead of the real module.
Fix: use monkeypatch.setitem() which auto-restores after the test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Harshit Jain
|
13130ea3e1
|
Litellm fix langfuse otel trace (#20382)
* fix: support multi-project keys and fix trace leakage * fix: Langfuse otel handle * fix lint errors mypy * passing all test case --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> |
||
|
Krish Dholakia
|
74ae7aed44
|
build: Squashed commit of the following: (#16176)
commit |
||
|
Katsuhiro Muto
|
99775fa0f8
|
Support responses API streaming in langfuse otel (#16153)
* streaming support in langfuse otel * Added testing for Langfuse Otel tracing in the response API --------- Co-authored-by: eycjur <eycjur@example.com> |
||
|
Katsuhiro Muto
|
2074b4d662
|
Fix: Support tool usage messages with Langfuse OTEL integration (#15932)
* Log tool use in langfuse otel integration * Add test for logging function calling --------- Co-authored-by: eycjur <eycjur@example.com> |
||
|
Mubashir Osmani
|
0fde408e35
|
added langfuse logging for responses api (#14597)
* added langfuse logging for responses api * tests added |
||
|
Ishaan Jaff
|
165242e31f
|
[Feat] langfuse_otel logger - allow using LANGFUSE_OTEL_HOST for configuring host (#14013)
* feat - add _get_langfuse_otel_host * test_get_langfuse_otel_config_with_otel_host_priority * docs: LANGFUSE_OTEL_HOST |
||
|
Ishaan Jaff
|
008ea864a7
|
[Feat] - Add key/team logging for Langfuse OTEL Logger (#13512)
* feat - add key/team logging for LF * test_construct_dynamic_otel_headers_with_langfuse_keys * update LangfuseOtelLogger * test_construct_dynamic_otel_headers_with_langfuse_keys * cleanup * OpenTelemetryConfig fixes * fix code qa checks * TestLangfuseOtelIntegration |
||
|
Alex Strick van Linschoten
|
75ae43e667
|
feat(langfuse-otel): Add comprehensive metadata support to Langfuse OpenTelemetry integration (#12956)
* feat(langfuse-otel): Add comprehensive metadata support to Langfuse OpenTelemetry integration
This commit brings the langfuse_otel integration to feature parity with the vanilla Langfuse integration by adding support for all metadata fields.
Changes:
- Extended LangfuseSpanAttributes enum with all supported metadata fields:
- Generation-level: generation_name, generation_id, parent_observation_id, version, mask_input/output
- Trace-level: trace_user_id, session_id, tags, trace_name, trace_id, trace_metadata, trace_version, trace_release, existing_trace_id, update_trace_keys
- Debug: debug_langfuse
- Implemented metadata extraction and mapping in langfuse_otel.py:
- Added _extract_langfuse_metadata() helper to extract metadata from kwargs
- Support for header-based metadata (langfuse_* headers) via proxy
- Enhanced _set_langfuse_specific_attributes() to map all metadata to OTEL attributes
- JSON serialization for complex types (lists, dicts) for OTEL compatibility
- Updated documentation:
- Added 'Metadata Support' section explaining all fields are now supported
- Provided usage example showing how to pass metadata
- Clarified that traces are viewed in Langfuse UI (not generic OTEL backends)
- Added opentelemetry-exporter-otlp to required dependencies
This allows users to pass metadata like:
metadata={
'generation_name': 'my-generation',
'trace_id': 'trace-123',
'session_id': 'session-456',
'tags': ['prod', 'v1'],
'trace_metadata': {'user_type': 'premium'}
}
All metadata is exported as OpenTelemetry span attributes with 'langfuse.*' prefix for easy filtering and analysis in the Langfuse UI.
* Fix ruff linting error
* test(langfuse-otel): Fix failing test and add comprehensive metadata tests
- Fix test_set_langfuse_environment_attribute to use positional arguments
instead of keyword arguments when asserting safe_set_attribute calls
- Add test_extract_langfuse_metadata_basic to verify metadata extraction
from litellm_params
- Add test_extract_langfuse_metadata_with_header_enrichment to test
integration with header-based metadata using a stubbed LangFuseLogger
- Add test_set_langfuse_specific_attributes_full_mapping to comprehensively
test all metadata field mappings and JSON serialization of complex types
These tests ensure full coverage of the langfuse_otel metadata features
added in commit
|
||
|
Ishaan Jaff
|
08f3b06d82
|
[Feat] Bump langfuse python SDK version and LANGFUSE_TRACING_ENVIRONMENT (#12376)
* bump langfuse to 2.59.7 * _set_langfuse_specific_attributes * fix files * test_set_langfuse_environment_attribute |
||
|
Krish Dholakia
|
308e82d885
|
LiteLLM SDK <-> Proxy improvement (don't transform message client-side) + Bedrock - handle qs:.. in base64 file data + Tag Management - support adding public model names (#11908)
* fix(factory.py): handle qs:.. in mime type Fixes https://github.com/BerriAI/litellm/issues/11839 * feat(litellm_proxy/): don't transform messages client-side leave litellm proxy messages untouched - allow proxy to handle transformation prevents double transformation * feat(tag_management_endpoints.py): support adding models to tag by adding model_name Closes https://github.com/BerriAI/litellm/issues/11884 * test(test_tag_management_endpoints.py): add unit tests for adding new model by public model name * test: update test |
Renamed from tests/litellm/integrations/test_langfuse_otel.py (Browse further)