Commit graph

47 commits

Author SHA1 Message Date
yucheng-berri
14338a2471
fix(redis): only use SSLConnection when ssl is truthy in connection pool kwargs (#32825) 2026-07-10 17:16:10 -07:00
Yassin Kortam
3ea7f98725
feat(proxy): configure the coordination redis independently of the response cache (#32661)
* fix(proxy): build redis usage cache from REDIS_* env when cache backend is not Redis

Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.

Resolves LIT-3861

* feat(proxy): configure the coordination redis independently of the response cache

Adds general_settings.coordination_redis, an explicit block for the Redis
the proxy uses for cross-pod rate limits, parallel-request limits, spend
tracking, the pod lock manager, and shared health checks. Resolution order
is the explicit block, then a plain-Redis response-cache backend, then the
REDIS_* environment. Cluster and sentinel targets are supported, and a
cluster target now builds a RedisClusterCache so cluster-aware consumers
take the cluster path.

Admins can configure it from the Caching page of the dashboard via
/coordination_redis/settings, which reports which source is in effect,
redacts credentials on read, and offers a connection test. Settings saved
there are read back at startup so they take effect on restart.

Also fixes redis client construction so an explicitly configured host
outranks REDIS_URL in the environment. Previously the url branch stripped
the caller's host and port, so an explicit block, or a connection test
typed into the dashboard, silently targeted whatever REDIS_URL named

* fix(ui): move coordination_redis_settings into renamed _components directory

---------

Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
2026-07-10 16:15:59 -07:00
Yassin Kortam
be4d0d8439
fix(redis): re-establish async cluster connections after a node restart (#31577)
Some checks failed
GitHub Actions Security Analysis / zizmor (push) Waiting to run
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
When redis_startup_nodes is set the async cluster client was built with no health check and no TCP keepalive, so a connection silently dropped by a cluster restart (e.g. ElastiCache Serverless maintenance) stayed in the pool and got reused while dead; the first command after the restart stalled in re-initialization until the LoggingWorker timeout cancelled it, surfacing as CancelledError then TimeoutError on the spend-counter path

Build the async cluster client with a 25s health_check_interval and socket_keepalive so an idle connection is PING-validated and reconnected before reuse, and expose both through the cluster kwarg allow-list so an explicit value from config still wins

Resolves LIT-4083
2026-06-30 12:25:15 -07:00
Mateo Wang
48b5a5a0cc
style: unify ruff format width on 120 (#31518)
The repo linted at 120 (E501, isort) but ran ruff format at 88 via a
--line-length 88 override in the Makefile and CI, leaving the formatter
and the linter disagreeing on wrap width. Drop the override so ruff.toml's
line-length = 120 is the single source of truth and reformat the tree to
match.
2026-06-27 12:39:29 -07:00
Mateo Wang
17bfd415ae
chore: migrate Python formatter from black to ruff format (#31317) 2026-06-25 11:27:43 -07:00
ryan-crabbe-berri
5a62806fdc
chore(lint): remove PLR0915 too-many-statements ruff rule (#30574)
Drops PLR0915 from ruff's extend-select along with its per-file-ignores,
and strips the now-unused `# noqa: PLR0915` directives across the codebase
(RUF100 would otherwise flag them as unused). The C901 suppression that
shared a directive with PLR0915 in streaming_handler.py is preserved.
2026-06-16 16:52:49 -07:00
Mateo Wang
9c775e7304
ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516)
* ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules

Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default),
RUF009 (function-call-in-dataclass-default-argument), and RUF100
(unused-noqa) in ruff.toml, and clean up every resulting violation.

RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or
blanket noqas: blanket `# noqa` are now scoped to the rule they actually
suppress (mostly T201), dead directives are removed, and inapplicable
codes are trimmed (e.g. F401 dropped from `import *`).

lint.external lists rules enforced outside this config (the strict-rule
gate via ruff-strict.toml and upstream litellm's own ruff config) so
RUF100 keeps the noqa directives that protect them instead of stripping
coverage this config can't see.

* ci(lint): trim RUF100 external list to load-bearing codes only

Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006,
PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in
the gated source. Keep only the 11 codes with live suppressions so
RUF100 doesn't flag them as unused. Future strict-gate suppressions can
re-add codes here (or fix the underlying issue) as needed.
2026-06-16 12:07:22 -07:00
Sameer Kankute
36c494fdd2
Litellm oss staging (#28161)
* fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455)

Squash-merged by litellm-agent from Anai-Guo's PR.

* feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508)

Squash-merged by litellm-agent from yimao's PR.

* fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550)

Squash-merged by litellm-agent from krisxia0506's PR.

* fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711)

Squash-merged by litellm-agent from krisxia0506's PR.

* fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503)

Squash-merged by litellm-agent from krisxia0506's PR.

* Fix Gemini MIME detection for extensionless GCS URIs (#27278)

Squash-merged by litellm-agent from krisxia0506's PR.

* fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107)

Squash-merged by litellm-agent from voidborne-d's PR.

* feat(chart): add support for autoscaling behavior in HPA (#27990)

Squash-merged by litellm-agent from FabrizioCafolla's PR.

* feat(proxy): add blocked flag to models for pause/resume from the UI (#27927)

Squash-merged by litellm-agent from Cyberfilo's PR.

* fix: pass socket timeouts to Redis cluster clients (#27920)

Squash-merged by litellm-agent from tomdee's PR.

* Fix/cache token (#28009)

Squash-merged by litellm-agent from escon1004's PR.

* fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080)

Squash-merged by litellm-agent from Divyansh8321's PR.

* fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617)

* fix: reset org and tag budgets (#27326)

* reset org budgets

* reset tag budgets

---------

Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>

* fix(ui): omit allowed_routes from key edit save when unchanged (#27553)

* fix(ui): omit allowed_routes from key edit save when unchanged

When a team admin opens Edit Settings on a key with key_type=AI APIs and
saves without changing anything, the UI re-sends the existing allowed_routes
value, which the backend's _check_allowed_routes_caller_permission gate
rejects for non-proxy-admins (LIT-2681).

Strip allowed_routes from the patch in handleSubmit when it deep-equals the
original keyData.allowed_routes. The backend treats absence as "leave alone,"
so no-op saves now succeed for non-admins. Admins explicitly editing the
field still send the new value.

* fix(ui): order-insensitive allowed_routes diff + cover null-original case

Address Greptile review:

- Switch the "is allowed_routes unchanged" check to a Set-based comparison so
  a server-side reorder of the array doesn't register as a user edit and
  re-trigger LIT-2681.
- Add two regression tests: (1) keyData.allowed_routes is null and the form
  is untouched — patch should strip the field; (2) server returned routes in
  a different order than the user originally entered — patch should still
  recognize the value as unchanged.

* chore(ui): strip ticket refs and tighten comments in key edit fix

- Remove internal-tracker references from in-code comments
- Tighten the WHY comment in handleSubmit to two lines
- Drop redundant test-block comments — test names already describe the case

* fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc

* fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests

GuardrailRaisedException and BlockedPiiEntityError both lacked a
status_code attribute.  When these exceptions reached the proxy
exception handler (getattr(e, 'status_code', 500)), the fallback
defaulted to HTTP 500 — making intentional guardrail blocks
indistinguishable from server errors and causing unnecessary client
retries.

Changes:
- Add status_code=400 (keyword-only) to GuardrailRaisedException
- Add status_code=400 (keyword-only) to BlockedPiiEntityError
- Update _is_guardrail_intervention() to recognize both exceptions
  so downstream loggers record 'guardrail_intervened' instead of
  'guardrail_failed_to_respond'
- Add 6 unit tests for default/custom status codes and getattr pattern
- Strengthen existing blocked-action test with status_code assertion

Fixes #24348

---------

Co-authored-by: Michael-RZ-Berri <michael@berri.ai>
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>

* fix(router/proxy): address Greptile P1+P2 review comments on PR #28161

- router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429)
  when a specifically-addressed deployment is administratively blocked; 429 misleads
  retry-enabled clients into spinning forever against a paused model
- proxy_server: compute get_fully_blocked_model_names() once before both branches in
  model_list() instead of duplicating the call in each branch
- deepseek: upgrade silent debug log to warning when injecting placeholder
  reasoning_content so callers are clearly notified of degraded multi-turn quality
- tests: update two blocked-deployment assertions to expect ServiceUnavailableError

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address bug detection findings (cache token order, mutable defaults)

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: address bugs in async pass-through, anthropic cache token detection, rerank tests

- async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments
- cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0
- dashscope rerank tests: pass request to httpx.Response constructions for consistency

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix code qa

* fix(vertex_ai/gemini): strip MIME parameters from GCS contentType

GCS object metadata's contentType field can include parameters such as
'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases
so downstream get_file_extension_from_mime_type sees a bare MIME type.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(vertex_ai/gemini): clarify mime-type error message string concatenation

Co-authored-by: Yassin Kortam <yassin@berri.ai>

---------

Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Vincent <yimao1231@gmail.com>
Co-authored-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: d 🔹 <liusway405@gmail.com>
Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Tom Denham <tom@tomdee.co.uk>
Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com>
Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com>
Co-authored-by: robin-fiddler <robin@fiddler.ai>
Co-authored-by: Michael-RZ-Berri <michael@berri.ai>
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
2026-05-18 16:27:44 -07:00
Krrish Dholakia
06c1c3497f
fix: Fix Redis Sentinel client handling to solve authentication error… (#26302)
* fix: Fix Redis Sentinel client handling to solve authentication error with password protected sentinel (#25625)

* fix Redis Sentinel authentication handling

* test: cover Redis Sentinel auth routing

* refactor: align Redis Sentinel kwargs threading

* fix: avoid duplicate Redis Sentinel socket timeouts

* Address review comments

* refactor(_redis): return set from _get_redis_kwargs for O(1) lookup

Align _get_redis_kwargs() with the cluster helper by returning a set
instead of a list, so the sentinel connection-kwargs filter uses O(1)
membership tests. Addresses Greptile review feedback on PR #26302.

* fix(_redis): restore Azure-specific kwargs in cluster kwargs set

The set-literal refactor of _get_redis_cluster_kwargs dropped four
LiteLLM-custom Azure keys (azure_redis_ad_token, azure_client_id,
azure_tenant_id, azure_client_secret) that the prior list form had
explicitly appended. Because they are not in RedisCluster's argspec,
they were silently stripped, breaking Azure IAM auth on cluster
clients. Re-add them to the explicit include set.

---------

Co-authored-by: Kristin Cowalcijk <kristincowalcijk@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: krrish-berri-2 <krrish-berri-2@users.noreply.github.com>
Co-authored-by: claude <claude@anthropic.com>
2026-05-13 14:06:58 -07:00
Shivam Rawat
c7739c9ed5
feat: add ability to auth to azure with token (#27556)
Squash-merged by litellm-agent from shivamrawat1's PR.
2026-05-09 22:34:09 +00:00
ishaan-berri
693ad49719
Litellm ishaan march23 - MCP Toolsets + GCP Caching fix (#25146) (#25155)
* Litellm ishaan march23 - MCP Toolsets + GCP Caching fix  (#25146)

* feat(mcp): MCP Toolsets — curated tool subsets from one or more MCP servers (#24335)

* feat(mcp): add LiteLLM_MCPToolsetTable and mcp_toolsets to ObjectPermissionTable

* feat(mcp): add prisma migration for MCPToolset table

* feat(mcp): add MCPToolset Python types

* feat(mcp): add toolset_db.py with CRUD helpers for MCPToolset

* feat(mcp): add toolset CRUD endpoints to mcp_management_endpoints

* fix(mcp): skip allow_all_keys servers when explicit mcp_servers permission is set (toolset scope fix)

* feat(mcp): add _apply_toolset_scope and toolset route handling in server.py

* fix(mcp): resolve toolset names in responses API before fetching tools

* feat(mcp): add mcp_toolsets field to LiteLLM_ObjectPermissionTable type

* feat(mcp): register LiteLLM_MCPToolsetTable in prisma client initialization

* feat(mcp): validate mcp_toolsets in key-vs-team permission check

* feat(mcp): register toolset routes in proxy_server.py

* feat(mcp): add MCPToolset and MCPToolsetTool TypeScript types

* feat(mcp): add fetchMCPToolsets, createMCPToolset, updateMCPToolset, deleteMCPToolset API functions

* feat(mcp): add useMCPToolsets React Query hook

* feat(mcp): add toolsets (purple) as third option type in MCPServerSelector

* feat(mcp): extract toolsets from combined MCP field in key form

* feat(mcp): extract toolsets from combined MCP field in team form

* feat(mcp): show toolsets section in MCPServerPermissions read view

* feat(mcp): pass mcp_toolsets through object_permissions_view

* feat(mcp): add MCPToolsetsTab component for creating and managing toolsets

* feat(mcp): add Toolsets tab to mcp_servers.tsx

* feat(mcp): pass mcpToolsets to playground chat and responses API calls

* feat(mcp): generate correct server_url for toolsets in playground API calls

* docs(mcp): add MCP Toolsets documentation

* docs(mcp): add mcp_toolsets to sidebar

* fix(mcp): replace x-mcp-toolset-id header with ContextVar to prevent client forgery

* fix(mcp): use ContextVar + StreamingResponse for toolset MCP routes (fixes SSE streaming)

* fix(mcp): cache toolset permission lookups to avoid per-request DB calls

* test(mcp): add tests for toolset scope enforcement, ContextVar isolation, and access control

* fix(mcp): cache toolset name lookups in MCPServerManager to avoid per-request DB calls

* fix(mcp): prevent body_iter deadlock + use cached toolset lookup in responses API

- _stream_mcp_asgi_response: add done callback to handler_task that puts
  the EOF sentinel on body_queue when the task exits, preventing body_iter
  from hanging forever if the handler raises after headers are sent.
- litellm_proxy_mcp_handler: replace raw get_mcp_toolset_by_name() DB call
  with global_mcp_server_manager.get_toolset_by_name_cached() so toolset
  resolution uses the 60s TTL cache added for this purpose instead of
  hitting the DB on every responses-API request.

* fix(mcp): toolset access control, asyncio fix, and real unit tests

- server.py: _apply_toolset_scope now enforces that non-admin keys must
  have the requested toolset_id in their mcp_toolsets grant list;
  admin keys always bypass the check.
- mcp_management_endpoints.py: three access-control fixes:
  * fetch_mcp_toolsets: non-admin keys with mcp_toolsets=None now
    return [] instead of all toolsets (only admins get 'all' when
    the field is absent)
  * fetch_mcp_toolset: non-admin keys that haven't been granted the
    requested toolset_id now get 403 instead of the full result
  * add_mcp_toolset: duplicate toolset_name now returns 409 Conflict
    instead of an opaque 500
- proxy_server.py: use asyncio.get_running_loop() instead of
  get_event_loop() inside an already-running coroutine (Python 3.10+).
- test_mcp_toolset_scope.py: replace four hollow tests that only
  asserted local variable properties with real tests that call the
  production fetch_mcp_toolsets() and handle_streamable_http_mcp()
  functions with mocked dependencies.

* fix(mcp): add mcp_toolsets to ObjectPermissionBase, fix multi-toolset overwrite, fix delete 404, allow standalone key toolsets

* fix(mcp): add auth check on toolset resolution in responses API; union mcp_servers in _merge_toolset_permissions

* fix(mcp): handle RecordNotFoundError in update_mcp_toolset; union direct servers with toolset servers

* fix(mcp): use _user_has_admin_view; deny None mcp_toolsets for non-admin; use direct RecordNotFoundError import; fix docstring

* fix(mcp): add @default(now()) to MCPToolsetTable.updated_at; fix test for non-admin toolset access

* fix: use UniqueViolationError import; guard _ensure_eof for error/cancel only

* fix(mcp): preserve mcp_access_groups in toolset scope, use shared Redis cache for toolset perms

- Remove mcp_access_groups=[] from _apply_toolset_scope (server.py) and the
  responses API toolset path (litellm_proxy_mcp_handler.py). A key's access-group
  grants remain valid even when the request is scoped to a single toolset; clearing
  them silently revoked legitimate entitlements.

- Switch resolve_toolset_tool_permissions and get_toolset_by_name_cached to use
  user_api_key_cache (Redis-backed DualCache in production) instead of per-instance
  in-memory dicts. Cache entries are now shared across workers, eliminating the
  per-worker stale-toolset-permission window flagged as a P1 by Greptile.

- Use union merge (set union of tool names per server) when applying toolset
  permissions in the responses API path so direct-server tool restrictions are not
  overwritten by toolset permissions.

* fix(mcp): return 404 when edit_mcp_toolset target does not exist

* fix(mcp): align mcp_toolsets default to None in LiteLLM_ObjectPermissionTable

* fix(mcp): admin toolset visibility, in-place tool name mutation, test helper coercion

* fix(mcp): treat None/[] team mcp_toolsets as no restriction in key validation

* fix(mcp): allow_all_keys backward compat, blocked_tools API write-path, efficient startup query

* fix(mcp): use _mcp_active_toolset_id ContextVar to detect toolset scope, avoiding DB-default false-positive

* fix(mcp): remove dead toolset cache stubs, log invalidation failures, align schema updated_at defaults

* fix(mcp): deserialise MCPToolset from Redis cache hit, replace fastapi import in test

* fix(mcp): evict name-cache on toolset mutation, 409 on rename conflict, warning-level list errors

* fix(redis): regenerate GCP IAM token per connection for async cluster (#24426)

* fix(redis): regenerate GCP IAM token per connection for async cluster clients

Async RedisCluster was generating the IAM token once at startup and
storing it as a static password. After the 1-hour GCP token TTL, any
new connection (including to newly-discovered cluster nodes) would fail
to authenticate.

Fix: introduce GCPIAMCredentialProvider that implements redis-py's
CredentialProvider protocol. It calls _generate_gcp_iam_access_token()
on every new connection, matching what the sync redis_connect_func
already does. async_redis.RedisCluster accepts a credential_provider
kwarg which is invoked per-connection.

* refactor(redis): move GCPIAMCredentialProvider to its own file

Extract GCPIAMCredentialProvider and _generate_gcp_iam_access_token
into litellm/_redis_credential_provider.py. _redis.py imports them
from there, keeping the public API unchanged.

* fix: address Greptile review issues

- GCPIAMCredentialProvider now inherits from redis.credentials.CredentialProvider
  so redis-py's async path calls get_credentials_async() properly
- move _redis_credential_provider import to top of _redis.py (PEP 8)
- remove dead else-branch that silently no-oped (gcp_service_account from
  redis_kwargs.get() was always None since it's popped by _get_redis_client_logic)
- remove mid-function 'from litellm import get_secret_str' inline import
- remove unused 'call' import from test_redis.py

* chore: retrigger CI/review

* chore: sync schema.prisma copies from root

* chore: sync schema.prisma copies from root

* fix(proxy_server): use bounded asyncio.Queue with maxsize to prevent unbounded growth

* fix(a2a/pydantic_ai): make api_base Optional to match base class signature

* fix(a2a/pydantic_ai): make api_base Optional in handler and guard against None

* fix(mcp): remove unused get_all_mcp_servers import

* fix(mcp): remove unused MCPToolset import

* refactor(mcp): extract toolset permission logic to reduce statement count below PLR0915 limit

* fix(tests): update reload_servers_from_database tests to mock prisma directly

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix(toolset_db): lazy-import prisma to avoid ImportError when prisma not installed

* fix(tests): update UI tests for toolset tab and updated empty state text

* fix(tests): add get_mcp_server_by_name to fake_manager stub

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-04 16:23:21 -07:00
cohml
3eeb14bf1a
fix(cache): Fix Redis cluster caching (#23480)
* fix redis cluster startup_nodes check order

* add tests for redis cluster startup_nodes fix
2026-03-17 08:32:01 -07:00
Chesars
2e9f057fbd style: run black formatter on entire codebase 2026-03-11 17:07:57 -03:00
Ryan Crabbe
6931fea929 fix: close leaked Redis connection pools on cache eviction and disconnect
- RC1: Override _remove_key() in LLMClientCache to schedule aclose() on
  evicted async clients instead of relying on GC
- RC2: Use passed connection_pool for URL configs instead of creating an
  orphaned pool via from_url()
- RC3: Pass max_connections through to BlockingConnectionPool.from_url()
  for URL configs, with input validation for invalid values
- RC5: Close sync redis_client in disconnect() with try/except guard
2026-02-20 17:09:32 -08:00
soo-jin.kim
03e1d93199
fix: Apply max_connections configuration to Redis async client (#15797)
* fix: Apply max_connections configuration to Redis async client

- Add max_connections to available Redis cluster kwargs
- Add connection_pool parameter to get_redis_async_client()
- Pass connection_pool to Redis client if provided
- Prevents Redis connection exhaustion under high load

* test: Add tests for Redis max_connections feature

- Test max_connections is included in cluster kwargs
- Test connection_pool parameter is properly passed to async client
- Test async client works without connection_pool parameter

All 3 tests pass successfully
2025-10-22 09:19:08 -07:00
Pierre-Emmanuel MERCIER
861750790b feat: add redis ssl and username support (#11319) 2025-10-08 16:48:07 -07:00
Ishaan Jaff
555efd60b5
[Security] Fix: Ensure .info() logs are not used for request/responses + Add code QA check for possible violations (#14386)
* test: info log

* fixes for debug

* add info_log_check

* fix: debug level

* fix: debug level

* fix: use .debug

* fix info statements

* fix info log check

* fix: fix emit_standard_logging_payload

* fix RequestBody

* Potential fix for code scanning alert no. 3339: Clear-text logging of sensitive information

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix check

* fix code qa check

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-09-09 13:55:56 -07:00
Ishaan Jaff
8920a645e9
Revert "feat: add redis ssl and username support (#11319)" (#14303)
This reverts commit 31f806f7d0.
2025-09-06 14:01:00 -07:00
Pierre-Emmanuel MERCIER
31f806f7d0
feat: add redis ssl and username support (#11319) 2025-09-05 10:35:11 -07:00
Jugal D. Bhatt
29a8c583c2
added redis iam auth (#13275) 2025-08-05 10:56:34 -07:00
Ishaan Jaff
3c43b0f8ad
add _pretty_print_redis_config (#12073) 2025-06-26 10:26:57 -07:00
Krish Dholakia
8ee32291e0
Squashed commit of the following: (#9709)
commit b12a9892b7
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Wed Apr 2 08:09:56 2025 -0700

    fix(utils.py): don't modify openai_token_counter

commit 294de31803
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 21:22:40 2025 -0700

    fix: fix linting error

commit cb6e9fbe40
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 19:52:45 2025 -0700

    refactor: complete migration

commit bfc159172d
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 19:09:59 2025 -0700

    refactor: refactor more constants

commit 43ffb6a558
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:45:24 2025 -0700

    fix: test

commit 04dbe4310c
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:28:58 2025 -0700

    refactor: refactor: move more constants into constants.py

commit 3c26284aff
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:14:46 2025 -0700

    refactor: migrate hardcoded constants out of __init__.py

commit c11e0de69d
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:11:21 2025 -0700

    build: migrate all constants into constants.py

commit 7882bdc787
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:07:37 2025 -0700

    build: initial test banning hardcoded numbers in repo
2025-04-02 21:24:54 -07:00
Ishaan Jaff
3d0d5e1473 run black formatting 2025-03-31 16:12:31 -07:00
jmarshall-medallia
da953c74a1
Add password to sync sentinel client (#9622) 2025-03-29 23:38:54 -07:00
Krrish Dholakia
db92956ae3 fix(redis_cache.py): add 5s default timeout 2025-03-17 14:27:36 -07:00
Ishaan Jaff
40e3af0428
(Redis Cluster) - Fixes for using redis cluster + pipeline (#8442)
* update RedisCluster creation

* update RedisClusterCache

* add redis ClusterCache

* update async_set_cache_pipeline

* cleanup redis cluster usage

* fix redis pipeline

* test_init_async_client_returns_same_instance

* fix redis cluster

* update mypy_path

* fix init_redis_cluster

* remove stub

* test redis commit

* ClusterPipeline

* fix import

* RedisCluster import

* fix redis cluster

* Potential fix for code scanning alert no. 2129: Clear-text logging of sensitive information

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix naming of redis cluster integration

* test_redis_caching_ttl_pipeline

* fix async_set_cache_pipeline

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-02-12 18:01:32 -08:00
Ishaan Jaff
c7f14e936a
(code quality) run ruff rule to ban unused imports (#7313)
* remove unused imports

* fix AmazonConverseConfig

* fix test

* fix import

* ruff check fixes

* test fixes

* fix testing

* fix imports
2024-12-19 12:33:42 -08:00
Ishaan Jaff
5c854650c2
(redis fix) - fix AbstractConnection.__init__() got an unexpected keyword argument 'ssl' (#6908)
* add better debugging for get_redis_connection_pool + allow passing ssl=None

* test_redis_with_ssl

* test_redis_with_ssl

* test_redis_with_ssl
2024-11-25 22:52:44 -08:00
Ishaan Jaff
d136641954
(fix proxy redis) Add redis sentinel support (#6154)
* add sentinel_password support

* add doc for setting redis sentinel password

* fix redis sentinel - use sentinel password
2024-11-12 18:36:46 -08:00
Krish Dholakia
cb2563e3c0
Litellm dev 10 22 2024 (#6384)
* fix(utils.py): add 'disallowed_special' for token counting on .encode()

Fixes error when '<
endoftext
>' in string

* Revert "(fix) standard logging metadata + add unit testing  (#6366)" (#6381)

This reverts commit 8359cb6fa9.

* add new 35 mode lcard (#6378)

* Add claude 3 5 sonnet 20241022 models for all provides (#6380)

* Add Claude 3.5 v2 on Amazon Bedrock and Vertex AI.

* added anthropic/claude-3-5-sonnet-20241022

* add new 35 mode lcard

---------

Co-authored-by: Paul Gauthier <paul@paulg.com>
Co-authored-by: lowjiansheng <15527690+lowjiansheng@users.noreply.github.com>

* test(skip-flaky-google-context-caching-test): google is not reliable. their sample code is also not working

* Fix metadata being overwritten in speech() (#6295)

* fix: adding missing redis cluster kwargs (#6318)

Co-authored-by: Ali Arian <ali.arian@breadfinancial.com>

* Add support for `max_completion_tokens` in Azure OpenAI (#6376)

Now that Azure supports `max_completion_tokens`, no need for special handling for this param and let it pass thru. More details: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure#api-support

* build(model_prices_and_context_window.json): add voyage-finance-2 pricing

Closes https://github.com/BerriAI/litellm/issues/6371

* build(model_prices_and_context_window.json): fix llama3.1 pricing model name on map

Closes https://github.com/BerriAI/litellm/issues/6310

* feat(realtime_streaming.py): just log specific events

Closes https://github.com/BerriAI/litellm/issues/6267

* fix(utils.py): more robust checking if unmapped vertex anthropic model belongs to that family of models

Fixes https://github.com/BerriAI/litellm/issues/6383

* Fix Ollama stream handling for tool calls with None content (#6155)

* test(test_max_completions): update test now that azure supports 'max_completion_tokens'

* fix(handler.py): fix linting error

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Low Jian Sheng <15527690+lowjiansheng@users.noreply.github.com>
Co-authored-by: David Manouchehri <david.manouchehri@ai.moda>
Co-authored-by: Paul Gauthier <paul@paulg.com>
Co-authored-by: John HU <hszqqq12@gmail.com>
Co-authored-by: Ali Arian <113945203+ali-arian@users.noreply.github.com>
Co-authored-by: Ali Arian <ali.arian@breadfinancial.com>
Co-authored-by: Anand Taralika <46954145+taralika@users.noreply.github.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
2024-10-22 21:18:54 -07:00
Krish Dholakia
fac3b2ee42
Add pyright to ci/cd + Fix remaining type-checking errors (#6082)
* fix: fix type-checking errors

* fix: fix additional type-checking errors

* fix: additional type-checking error fixes

* fix: fix additional type-checking errors

* fix: additional type-check fixes

* fix: fix all type-checking errors + add pyright to ci/cd

* fix: fix incorrect import

* ci(config.yml): use mypy on ci/cd

* fix: fix type-checking errors in utils.py

* fix: fix all type-checking errors on main.py

* fix: fix mypy linting errors

* fix(anthropic/cost_calculator.py): fix linting errors

* fix: fix mypy linting errors

* fix: fix linting errors
2024-10-05 17:04:00 -04:00
Krish Dholakia
8039b95aaf
LiteLLM Minor Fixes & Improvements (09/21/2024) (#5819)
* fix(router.py): fix error message

* Litellm disable keys (#5814)

* build(schema.prisma): allow blocking/unblocking keys

Fixes https://github.com/BerriAI/litellm/issues/5328

* fix(key_management_endpoints.py): fix pop

* feat(auth_checks.py): allow admin to enable/disable virtual keys

Closes https://github.com/BerriAI/litellm/issues/5328

* docs(vertex.md): add auth section for vertex ai

Addresses - https://github.com/BerriAI/litellm/issues/5768#issuecomment-2365284223

* build(model_prices_and_context_window.json): show which models support prompt_caching

Closes https://github.com/BerriAI/litellm/issues/5776

* fix(router.py): allow setting default priority for requests

* fix(router.py): add 'retry-after' header for concurrent request limit errors

Fixes https://github.com/BerriAI/litellm/issues/5783

* fix(router.py): correctly raise and use retry-after header from azure+openai

Fixes https://github.com/BerriAI/litellm/issues/5783

* fix(user_api_key_auth.py): fix valid token being none

* fix(auth_checks.py): fix model dump for cache management object

* fix(user_api_key_auth.py): pass prisma_client to obj

* test(test_otel.py): update test for new key check

* test: fix test
2024-09-21 18:51:53 -07:00
Krish Dholakia
d46660ea0f
LiteLLM Minor Fixes & Improvements (09/18/2024) (#5772)
* fix(proxy_server.py): fix azure key vault logic to not require client id/secret

* feat(cost_calculator.py): support fireworks ai cost tracking

* build(docker-compose.yml): add lines for mounting config.yaml to docker compose

Closes https://github.com/BerriAI/litellm/issues/5739

* fix(input.md): update docs to clarify litellm supports content as a list of dictionaries

Fixes https://github.com/BerriAI/litellm/issues/5755

* fix(input.md): update input.md to include all message values

* fix(image_handling.py): follow image url redirects

Fixes https://github.com/BerriAI/litellm/issues/5763

* fix(router.py): Fix model key/base leak in error message

Fixes https://github.com/BerriAI/litellm/issues/5762

* fix(http_handler.py): fix linting error

* fix(azure.py): fix logging to show azure_ad_token being used

Fixes https://github.com/BerriAI/litellm/issues/5767

* fix(_redis.py): add redis sentinel support

Closes https://github.com/BerriAI/litellm/issues/4381

* feat(_redis.py): add redis sentinel support

Closes https://github.com/BerriAI/litellm/issues/4381

* test(test_completion_cost.py): fix test

* Databricks Integration: Integrate Databricks SDK as optional mechanism for fetching API base and token, if unspecified (#5746)

* LiteLLM Minor Fixes & Improvements (09/16/2024)  (#5723)

* coverage (#5713)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Move (#5714)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(litellm_logging.py): fix logging client re-init (#5710)

Fixes https://github.com/BerriAI/litellm/issues/5695

* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config

Fixes https://github.com/BerriAI/litellm/issues/5682

* feat(o1_handler.py): fake streaming for openai o1 models

Fixes https://github.com/BerriAI/litellm/issues/5694

* docs: deprecated traceloop integration in favor of native otel (#5249)

* fix: fix linting errors

* fix: fix linting errors

* fix(main.py): fix o1 import

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730)

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view

Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it

* fix(custom_logger.py): reset calltype

* fix: fix linting errors

* fix: fix linting error

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix: fix import

* Fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* DB test

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Coverage

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* progress

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix test name

Signed-off-by: dbczumar <corey.zumar@databricks.com>

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* test: fix test

* test(test_databricks.py): fix test

* fix(databricks/chat.py): handle custom endpoint (e.g. sagemaker)

* Apply code scanning fix for clear-text logging of sensitive information

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(__init__.py): fix known fireworks ai models

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2024-09-19 13:25:29 -07:00
Ishaan Jaff
9a9c0e42eb allow setting password for redis cluster 2024-09-07 09:25:13 -07:00
Ishaan Jaff
5c4f3a9a34 fix allow using .env vars for redis cluster 2024-09-07 08:54:40 -07:00
Ishaan Jaff
9225d31776 allow setting REDIS_CLUSTER_NODES in .env 2024-09-07 07:41:49 -07:00
Krrish Dholakia
e2d7539690 feat(caching.py): redis cluster support
Closes https://github.com/BerriAI/litellm/issues/4358
2024-08-21 15:01:52 -07:00
Krrish Dholakia
6575143460 feat(proxy_server.py): return litellm version in response headers 2024-05-08 16:00:08 -07:00
Krrish Dholakia
d1217b955a fix(_redis.py): support redis ssl as a kwarg REDIS_SSL 2024-04-20 10:19:44 -07:00
Krrish Dholakia
84685b5f34 fix(_redis.py): fix args passed to redis.from_url argument 2024-04-19 15:27:11 -07:00
Krrish Dholakia
66c7e35a97 fix(redis.py): fix instantiating redis client from url 2024-02-15 17:48:00 -08:00
Krrish Dholakia
01df37d8cf fix(caching.py): use bulk writes and blockconnectionpooling for reads from Redis 2024-01-13 11:50:50 +05:30
Krrish Dholakia
007870390d fix: support async redis caching 2024-01-12 21:46:41 +05:30
Krrish Dholakia
4905929de3 refactor: add black formatting 2023-12-25 14:11:20 +05:30
Krrish Dholakia
91ce50338a fix(_redis.py): check if string before checking os.environ 2023-12-07 15:08:11 -08:00
Krrish Dholakia
e5638e2c5d fix(router.py): fix default caching response value 2023-12-07 13:44:31 -08:00
Krrish Dholakia
88c95ca259 fix(_redis.py): support additional params for redis 2023-12-05 12:16:51 -08:00