mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
1c61c2606e
* perf(auth): prefetch user, team, membership, org and project in one MGET, one query and one pipeline Auth read each object with its own Redis GET and, on a miss, its own DB query, then the admission spend counters with one GET each. The prefetch warms every entry the checks read with one MGET, one raw query for the Redis misses and one pipeline write, and a per-request batch serves the spend counter reads from one MGET. The per-object getters stay the readers and the fallback, so enforcement does not depend on the prefetch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(auth): keep prefetch and spend batch collections immutable Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(auth): let the cold spend-counter reseed reuse the admission MGET instead of one GET per counter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(auth): prefetch referenced auth objects only after the key's model access check passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): give the prefetch-ordering test's patches their test-quality reasons Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): move the real-Postgres prefetch join test to the proxy_behavior shard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): read NULL nested permission and budget lists as [] in the prefetch join Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(proxy): batch post-call spend counter reads and carry budget state through the request Post-call warm checks, reservation reads and reconcile reads for one request now go through a task-local spend counter batch: one MGET answers every counter, successful increments write their result back into the batch so no second Redis read follows, and invalidation forgets the key. RedisCache.async_increment sends INCRBYFLOAT and its TTL command in one pipeline round trip. Auth pins frozen team, user and org budget snapshots on UserAPIKeyAuth, the pre-call setup writes them into the request metadata, and Prometheus reads them back instead of calling get_key_object, get_team_object, get_user_object and get_org_object on the response path. The getters stay as the fallback for requests that carried nothing (custom auth, unauthenticated routes, skipped checks). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(proxy): reconcile the budget reservation and the post-call warm checks from one MGET and one pipeline A scope opened inside an open spend counter batch binds into it instead of starting its own, so the reservation reconcile and the post-call warm checks share the request's single MGET. The reconcile reads every reserved counter concurrently, sends the consistent adjustments in one INCRBYFLOAT+EXPIRE pipeline and settles a flushed or reseeded counter on its own afterwards, keeping the pre-call resize fail-closed. PendingSpendIncrement moves to spend_counter_batch so budget_reservation can build a pipeline without importing a private name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(proxy): drop the dataclass import left behind by the PendingSpendIncrement move Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): import Self from typing_extensions so the proxy imports on Python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use a neutral organization alias in the carried budget state tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover recorded and forgotten spend counter values in the request batch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): assert async_set_cache_pipeline_with_ttls keeps per-entry TTLs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): type the reservation entry carried through reconcile adjustments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): map the model table's aliases column to model_aliases in the prefetch join and read user memberships the way get_user_object does Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| arize | ||
| azure_storage | ||
| bitbucket | ||
| cloudzero | ||
| code_interpreter_interception | ||
| compression_interception | ||
| datadog | ||
| dotprompt | ||
| focus | ||
| gcs_bucket | ||
| gcs_pubsub | ||
| gitlab | ||
| helicone | ||
| langfuse | ||
| levo | ||
| litellm_agent | ||
| mavvrik_focus | ||
| newrelic | ||
| open_telemetry | ||
| opik | ||
| otel | ||
| pointfive | ||
| SlackAlerting | ||
| vector_store_integrations | ||
| websearch_interception | ||
| rubrik_test_helpers.py | ||
| test_agentops.py | ||
| test_anthropic_cache_control_hook.py | ||
| test_athina.py | ||
| test_azure_sentinel.py | ||
| test_braintrust_logging.py | ||
| test_braintrust_span_name.py | ||
| test_custom_guardrail.py | ||
| test_custom_guardrail_recursion.py | ||
| test_custom_prompt_management.py | ||
| test_deepeval.py | ||
| test_galileo.py | ||
| test_guardrail_logging_sync.py | ||
| test_helicone.py | ||
| test_langfuse.py | ||
| test_langfuse_otel.py | ||
| test_langsmith_init.py | ||
| test_lunary.py | ||
| test_mlflow.py | ||
| test_openmeter.py | ||
| test_opentelemetry.py | ||
| test_opentelemetry_dynamic_imports.py | ||
| test_opik_utils.py | ||
| test_otel_guardrail_violation_spans.py | ||
| test_otel_team_attributes_matrix.py | ||
| test_prometheus_api_promql_escape.py | ||
| test_prometheus_budget_metric_guard.py | ||
| test_prometheus_budget_metrics_db_lookups.py | ||
| test_prometheus_budget_metrics_timeout.py | ||
| test_prometheus_cache_metrics.py | ||
| test_prometheus_caller_identity.py | ||
| test_prometheus_carried_budget_state.py | ||
| test_prometheus_client_ip_user_agent.py | ||
| test_prometheus_custom_metadata_label_counts.py | ||
| test_prometheus_end_user_cardinality.py | ||
| test_prometheus_input_sequence_length_label.py | ||
| test_prometheus_invalid_key_filtering.py | ||
| test_prometheus_labels.py | ||
| test_prometheus_mcp_tool_metrics.py | ||
| test_prometheus_media_generation_metrics.py | ||
| test_prometheus_metric_name_consistency.py | ||
| test_prometheus_metrics_endpoint.py | ||
| test_prometheus_missing_metrics.py | ||
| test_prometheus_none_metadata.py | ||
| test_prometheus_overhead_with_guardrails.py | ||
| test_prometheus_queue_guardrail_metrics.py | ||
| test_prometheus_rate_limit_labels.py | ||
| test_prometheus_remaining_tokens_router_fallback.py | ||
| test_prometheus_requested_model_cardinality.py | ||
| test_prometheus_service_tier_label.py | ||
| test_prometheus_services.py | ||
| test_prometheus_spend_logs_metadata.py | ||
| test_prometheus_stream_label.py | ||
| test_prometheus_token_detail_metrics.py | ||
| test_prometheus_user_team_metrics.py | ||
| test_prompt_manager_ssti.py | ||
| test_responses_background_cost.py | ||
| test_rubrik.py | ||
| test_s3.py | ||
| test_s3_v2.py | ||
| test_shadow_eval_logger.py | ||
| test_weave_otel.py | ||