scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-05 08:07:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 137

Fix greptile reviews and mock test

Browse source
This commit is contained in:
Sameer Kankute 2026-03-23 22:38:54 +05:30
parent 0450194474
commit 9262c0d7c2
4 changed files with 40 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
docs/my-website/docs/proxy/config_settings.md
View file

@ -804,6 +804,7 @@ router_settings:
| LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS | Optionally enable semantic logs for OTEL
| LITELLM_OTEL_INTEGRATION_ENABLE_METRICS | Optionally enable emantic metrics for OTEL
| LITELLM_ENABLE_PYROSCOPE | If true, enables Pyroscope CPU profiling. Profiles are sent to PYROSCOPE_SERVER_ADDRESS. Off by default. See [Pyroscope profiling](/proxy/pyroscope_profiling).
| LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS | When `true`, if a team's legacy `model_aliases` entry maps a public model name to an internal `model_name_<team_id>_<uuid>` deployment, pre-call handling can skip that rewrite when team-scoped sibling deployments exist for the public name—so load balancing / `order` apply across siblings. Default is `false` for backwards compatibility. See [Team-scoped models and legacy aliases](./load_balancing#team-scoped-models-and-legacy-model_aliases). When stale aliases are detected and this flag is off, the proxy may log a one-time warning.
| PYROSCOPE_APP_NAME | Application name reported to Pyroscope. Required when LITELLM_ENABLE_PYROSCOPE is true. No default.
| PYROSCOPE_SERVER_ADDRESS | Pyroscope server URL to send profiles to. Required when LITELLM_ENABLE_PYROSCOPE is true. No default.
| PYROSCOPE_SAMPLE_RATE | Optional. Sample rate for Pyroscope profiling (integer). No default; when unset, the pyroscope-io library default is used.

15
docs/my-website/docs/proxy/load_balancing.md
View file

@ -336,6 +336,21 @@ The `order` parameter requires `enable_pre_call_checks: true` in `router_setting
If `order=1` deployment is unavailable (e.g., rate-limited), the router falls back to `order=2` deployments.
### Team-scoped models and legacy `model_aliases` {#team-scoped-models-and-legacy-model_aliases}
Team-scoped deployments are identified by `model_info.team_id` and `model_info.team_public_model_name`. Requests should use the **public** model name; the router resolves all sibling deployments (same public name, different `api_base` / `order`, etc.) for routing, failover, and deployment `order`.
For router internals: when a `team_id` is in scope, optimized lookups key off `(team_id, team_public_model_name)`. If code passes an internal deployment id (e.g. `model_name_<team_id>_<uuid>`) instead of the public name, routing still works via the usual deployment-name paths, but the team-specific fast path applies only to the public name.
**Legacy teams:** Older proxy versions could persist `model_aliases` on the team row mapping a public name to a single internal deployment id (`model_name_<team_id>_<uuid>`). On each request, pre-call logic may still rewrite `model` to that internal name **before** routing, which collapses to one deployment and can make newer sibling deployments unreachable.
**Migration options:**
1. **Recommended for upgrades:** Set environment variable `LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS=true` so that when sibling team deployments exist for the public name, the stale alias rewrite is skipped and team-scoped routing (including `order` and failover) applies. See the [Environment variables](./config_settings) table in the proxy settings doc.
2. **Data cleanup:** Remove obsolete `model_aliases` entries for team public names from the team record in the database so only `team_public_model_name` + team model list drive access.
If a stale alias is detected and the bypass is **not** enabled, the proxy may emit a **one-time** warning in logs explaining that sibling deployments may be unreachable until the flag is set or aliases are cleaned up.
### When You'll See Load Balancing in Action
**Immediate Effects:**

23
litellm/proxy/litellm_pre_call_utils.py
View file

@ -37,6 +37,7 @@ from litellm.types.utils import (
)
service_logger_obj = ServiceLogging() # used for tracking latency on OTEL
_STALE_TEAM_ALIAS_WARNING_KEYS: set[str] = set()
if TYPE_CHECKING:
@ -1320,16 +1321,28 @@ def _update_model_if_team_alias_exists(
)
# Check if the alias points to a team-scoped UUID name
# (format: "model_name_{team_id}_{uuid}")
if enable_stale_alias_bypass and aliased_target.startswith(
is_stale_team_alias = aliased_target.startswith(
f"model_name_{user_api_key_dict.team_id}_"
):
)
if is_stale_team_alias and llm_router:
# This is a stale alias from pre-PR deployments.
# Check if current team deployments exist for the public name.
if llm_router:
key = (user_api_key_dict.team_id, _model)
if key in llm_router.team_model_to_deployment_indices:
key = (user_api_key_dict.team_id, _model)
if key in llm_router.team_model_to_deployment_indices:
if enable_stale_alias_bypass:
# Team deployments exist; skip stale alias
return
warning_key = f"{user_api_key_dict.team_id}:{_model}:{aliased_target}"
if warning_key not in _STALE_TEAM_ALIAS_WARNING_KEYS:
_STALE_TEAM_ALIAS_WARNING_KEYS.add(warning_key)
verbose_proxy_logger.warning(
"Stale team model alias detected for model='%s', team_id='%s'. "
"New sibling deployments may be unreachable. "
"Set LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS=true to enable "
"team-scoped sibling routing.",
_model,
user_api_key_dict.team_id,
)
data["model"] = aliased_target
return

6
litellm/router.py
View file

@ -8256,6 +8256,12 @@ class Router:
if team_id specified, only return team-specific models
Optimized with O(1) index lookup instead of O(n) linear scan.
Note: when team_id is provided, O(1) lookup in
`team_model_to_deployment_indices` only applies when `model_name` is the
team public model name. If a caller passes an internal deployment model
name (for example, `model_name_<team_id>_<uuid>`), this method falls back
to the standard model-name index / scan path.
"""
returned_models: List[DeploymentTypedDict] = []