mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
fix(proxy): surface runtime-registered callbacks in UI Logging page (#38974)
* fix(proxy): surface runtime-registered callbacks in /get/config/callbacks Config-file callbacks fire at runtime but never appear in the UI Logging and Alerts page because /get/config/callbacks only reads the DB-merged config. Append runtime-registered callbacks from LoggingCallbackManager as read-only rows, deduplicated against configured rows via alias normalization. UI hides edit/delete/test actions for read-only rows. * fix: filter internal proxy hooks from runtime callbacks, update test - Filter _PROXY*, ShadowEval, ServiceLogging, SkillsInjection, ResponsesID prefixes - Update test to exclude read_only rows from count assertions - Still allows deployment/guardrail callbacks to surface if configured Note: comprehensive internal-hook filtering deferred, live-pr-risk will observe real behavior on running proxy. * fix: guard non-list config callbacks in get_config, use monkeypatch in tests - Line-concat type error: normalize_callback now returns empty list for non-list types (dict/tuple/set) instead of passing through unchanged; prevents TypeError when config values are non-list - Test quality TQ005: replace manual try/finally save-restore of litellm.callbacks with monkeypatch.setattr in test_get_config_callbacks_appends_runtime_only_callbacks and test_get_config_callbacks_redacts_runtime_only_row_secrets_for_view_only_admin - Ruff format: wrap _internal_callback_prefixes tuple and isinstance check across multiple lines to respect 120-char limit - All three new tests pass * fix: rework runtime callback inventory filtering and dedup - Filter internal proxy hooks by name: _PROXY_ prefix plus fixed internal names (cache, _ProxyDBLogger, deployment callbacks, service hooks) - Hide guardrail instances and runtime instances of already configured callbacks via CustomLoggerRegistry class lookup - Sort runtime rows and dedup per mode for stable output - normalize_callback returns tuples for str/None/list config values and empty for any other type - Tests mock get_callbacks_by_type explicitly and pin the exact row set; UI test covers read_only action hiding * fix: list dict-shaped callback config values by their keys Dict-valued success_callback/failure_callback/callbacks settings previously listed their keys as editable rows; keep that behavior instead of dropping them to read-only runtime rows. Adds a pin test for the dict shape. * fix: mark dotted-path callbacks read-only to prevent duplicate display Configured callbacks loaded from dotted Python paths (e.g. custom_callbacks.my_logger) are never matched against runtime instances by name because the registry uses short canonical names (e.g. langsmith, arize). Mark these rows read-only to prevent the UI from attempting delete operations that would fail at the endpoint level anyway. * fix: dedupe dotted-path callbacks by instance module instead of marking them read-only A dotted-path callback loaded from config registers as an object, so it surfaces at runtime under its class name and never matched the configured string, producing a second row. Marking the config row read_only hid the duplicate but also hid delete, which does work for these rows. Match the live instance back to its configured entry by module and drop it from the runtime rows, so the callback stays a single editable row. * test: cover dotted-path dedup across success, failure, and callbacks modes * fix(proxy): filter runtime callback inventory by object identity and label read-only rows in the UI Runtime-only rows were filtered by callback name, which missed initialized CustomLogger instances, router and proxy hook methods, guardrails, and user functions. The inventory now inspects the live callback objects through a public LoggingCallbackManager.get_callback_objects accessor and hides litellm-internal hooks, guardrails, and instances of already configured callbacks. The dashboard shows a Read only label for runtime-only rows instead of an empty action cell Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): keep configured-callback assertions minimal when runtime rows are present Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): hide internal cache string callback from runtime callback inventory Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): hide auto-registered vector store hook from callback inventory Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep YAML OTel-family callbacks listed next to a configured one arize, weave_otel and langfuse_otel all initialize OpenTelemetry subclasses, so hiding runtime callbacks by configured class made one saved OTel callback swallow its YAML siblings. Match runtime instances by their own callback_name and only fall back to class identity for bare OpenTelemetry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover scalar and null YAML callback keys in callback inventory Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop docstrings that restate callback inventory helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep runtime-only s3 and sqs callbacks in UI Logging inventory _is_litellm_internal_callback checked registry membership with the display alias (s3, sqs), which is not a registry key, so runtime-only S3Logger and SQSLogger instances were classified as internal and dropped from /get/config/callbacks. Check the registered name instead and cover both loggers in the internal-exclusion regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
yucheng-berri
GitHub
parent
e11be9de21
commit
00b631883d
7 changed files with 576 additions and 123 deletions
|
|
@ -441,7 +441,15 @@ class LoggingCallbackManager:
|
|||
|
||||
return result
|
||||
|
||||
def get_callback_objects(self) -> tuple[tuple[str, CustomLogger | Callable], ...]:
|
||||
return tuple(
|
||||
(self._get_callback_string(callback), callback)
|
||||
for callback in self._get_all_callbacks()
|
||||
if not isinstance(callback, str)
|
||||
)
|
||||
|
||||
def _get_callback_string(self, callback: CustomLogger | Callable | str) -> str:
|
||||
from litellm.integrations.opentelemetry import OpenTelemetry
|
||||
from litellm.litellm_core_utils.custom_logger_registry import (
|
||||
CustomLoggerRegistry,
|
||||
)
|
||||
|
|
@ -449,6 +457,8 @@ class LoggingCallbackManager:
|
|||
"""Convert a callback to its string representation"""
|
||||
if isinstance(callback, str):
|
||||
return callback
|
||||
elif isinstance(callback, OpenTelemetry) and callback.callback_name is not None:
|
||||
return callback.callback_name
|
||||
elif isinstance(callback, CustomLogger):
|
||||
# Try to get the string representation from the registry
|
||||
callback_str: Final = CustomLoggerRegistry.get_callback_str_from_class_type(type(callback))
|
||||
|
|
|
|||
|
|
@ -268,7 +268,7 @@ from litellm.constants import (
|
|||
WEEKLY_SPEND_REPORT_JOB_ID,
|
||||
)
|
||||
from litellm.exceptions import RejectedRequestError
|
||||
from litellm.integrations.custom_guardrail import ModifyResponseException
|
||||
from litellm.integrations.custom_guardrail import CustomGuardrail, ModifyResponseException
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.integrations.SlackAlerting.slack_alerting import SlackAlerting
|
||||
from litellm.litellm_core_utils.agentic_loop_settings import (
|
||||
|
|
@ -17655,6 +17655,66 @@ async def delete_callback(
|
|||
)
|
||||
|
||||
|
||||
def _normalize_callback_alias(callback_name: str) -> str:
|
||||
callback_aliases: Final = (
|
||||
("opentelemetry", "otel"),
|
||||
("s3_v2", "s3"),
|
||||
("aws_sqs", "sqs"),
|
||||
("custom_callback_api", "generic_api"),
|
||||
)
|
||||
return next(
|
||||
(canonical_name for alias, canonical_name in callback_aliases if alias == callback_name),
|
||||
callback_name,
|
||||
)
|
||||
|
||||
|
||||
def _callback_module_name(callback: CustomLogger | Callable[..., object]) -> str:
|
||||
if inspect.ismethod(callback):
|
||||
return callback.__func__.__module__
|
||||
if inspect.isfunction(callback):
|
||||
return callback.__module__
|
||||
return type(callback).__module__
|
||||
|
||||
|
||||
def _is_litellm_internal_callback(callback_name: str, callback: CustomLogger | Callable[..., object]) -> bool:
|
||||
from litellm.litellm_core_utils.custom_logger_registry import CustomLoggerRegistry
|
||||
|
||||
module_owner: Final = _callback_module_name(callback).partition(".")[0]
|
||||
is_registered_integration: Final = callback_name in CustomLoggerRegistry.CALLBACK_CLASS_STR_TO_CLASS_TYPE
|
||||
return not is_registered_integration and module_owner in ("litellm", "litellm_enterprise")
|
||||
|
||||
|
||||
def _is_instance_of_configured_callback(
|
||||
callback_name: str, callback: CustomLogger | Callable[..., object], configured_classes: tuple[type, ...]
|
||||
) -> bool:
|
||||
"""Self-naming OTel-family instances (`arize`, `weave_otel`) match by name, so a configured `logfire` (a bare
|
||||
`OpenTelemetry`) does not hide YAML-configured siblings of the same class."""
|
||||
from litellm.litellm_core_utils.custom_logger_registry import CustomLoggerRegistry
|
||||
|
||||
class_derived_name: Final = CustomLoggerRegistry.get_callback_str_from_class_type(type(callback))
|
||||
return isinstance(callback, configured_classes) and callback_name in (class_derived_name, type(callback).__name__)
|
||||
|
||||
|
||||
def _hidden_runtime_callback_names(configured_callback_names: frozenset[str]) -> frozenset[str]:
|
||||
from litellm.litellm_core_utils.custom_logger_registry import CustomLoggerRegistry
|
||||
|
||||
configured_classes: Final = tuple(
|
||||
CustomLoggerRegistry.CALLBACK_CLASS_STR_TO_CLASS_TYPE[name]
|
||||
for name in configured_callback_names
|
||||
if name in CustomLoggerRegistry.CALLBACK_CLASS_STR_TO_CLASS_TYPE
|
||||
)
|
||||
configured_modules: Final = frozenset(name.rsplit(".", 1)[0] for name in configured_callback_names if "." in name)
|
||||
internal_callback_names: Final = frozenset({"cache", "vector_store_pre_call_hook"})
|
||||
return internal_callback_names | frozenset(
|
||||
callback_name
|
||||
for callback_name, callback in litellm.logging_callback_manager.get_callback_objects()
|
||||
if isinstance(callback, CustomGuardrail)
|
||||
or _is_litellm_internal_callback(callback_name, callback)
|
||||
or _is_instance_of_configured_callback(callback_name, callback, configured_classes)
|
||||
or _callback_module_name(callback) in configured_modules
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/get/config/callbacks",
|
||||
tags=["config.yaml"],
|
||||
|
|
@ -17687,10 +17747,10 @@ async def get_config(
|
|||
# Normalize string callbacks to lists
|
||||
def normalize_callback(callback):
|
||||
if isinstance(callback, str):
|
||||
return [callback]
|
||||
elif callback is None:
|
||||
return []
|
||||
return callback
|
||||
return (callback,)
|
||||
if callback is None:
|
||||
return ()
|
||||
return tuple(callback) if isinstance(callback, (list, dict)) else ()
|
||||
|
||||
_success_callbacks = normalize_callback(_success_callbacks)
|
||||
_failure_callbacks = normalize_callback(_failure_callbacks)
|
||||
|
|
@ -17721,6 +17781,30 @@ async def get_config(
|
|||
for _callback in _success_and_failure_callbacks:
|
||||
_data_to_return.append(process_callback(_callback, "success_and_failure", environment_variables))
|
||||
|
||||
configured_callback_names: Final = frozenset(
|
||||
_normalize_callback_alias(callback)
|
||||
for callback in (_success_callbacks + _failure_callbacks + _success_and_failure_callbacks)
|
||||
)
|
||||
runtime_callbacks_by_type: Final = litellm.logging_callback_manager.get_callbacks_by_type()
|
||||
hidden_callback_names: Final = _hidden_runtime_callback_names(configured_callback_names)
|
||||
runtime_callback_rows: Final = tuple(
|
||||
(_normalize_callback_alias(callback_name), callback_type)
|
||||
for callback_type, callback_names in (
|
||||
("success", runtime_callbacks_by_type["success"]),
|
||||
("failure", runtime_callbacks_by_type["failure"]),
|
||||
("success_and_failure", runtime_callbacks_by_type["success_and_failure"]),
|
||||
)
|
||||
for callback_name in callback_names
|
||||
if callback_name not in hidden_callback_names
|
||||
)
|
||||
runtime_only_rows: Final = sorted(
|
||||
frozenset(row for row in runtime_callback_rows if row[0] not in configured_callback_names)
|
||||
)
|
||||
_data_to_return.extend(
|
||||
dict(process_callback(callback_name, callback_type, environment_variables), read_only=True)
|
||||
for callback_name, callback_type in runtime_only_rows
|
||||
)
|
||||
|
||||
_data_to_return = _apply_callback_role_gate(_data_to_return, is_full_admin)
|
||||
|
||||
# Check if slack alerting is on
|
||||
|
|
|
|||
|
|
@ -2920,7 +2920,7 @@ async def test_get_config_callbacks_with_all_types(client_no_auth):
|
|||
assert result["status"] == "success"
|
||||
assert "callbacks" in result
|
||||
|
||||
callbacks = result["callbacks"]
|
||||
callbacks = [cb for cb in result["callbacks"] if not cb.get("read_only", False)]
|
||||
|
||||
# Verify we have all 5 callbacks (2 success + 1 failure + 2 success_and_failure)
|
||||
assert len(callbacks) == 5
|
||||
|
|
|
|||
|
|
@ -13,9 +13,12 @@ Routes covered:
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from .conftest import VOLATILE_KEYS, normalize
|
||||
|
||||
|
||||
|
|
@ -229,10 +232,7 @@ def test_config_update_no_db_error(client, auth_as, monkeypatch):
|
|||
json={"general_settings": {"alerting": ["slack"]}},
|
||||
)
|
||||
assert response.status_code != 200
|
||||
assert (
|
||||
"db" in str(response.json()).lower()
|
||||
or "connect" in str(response.json()).lower()
|
||||
)
|
||||
assert "db" in str(response.json()).lower() or "connect" in str(response.json()).lower()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -273,9 +273,7 @@ def test_config_field_update_happy_admin(client, auth_as, mock_prisma, monkeypat
|
|||
}
|
||||
|
||||
|
||||
def test_config_field_update_non_admin_rejected(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_update_non_admin_rejected(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""Non-admin cannot update config fields — returns 400 with not-allowed
|
||||
detail (handler uses 400 for the auth gate, not 403)."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
|
|
@ -335,9 +333,7 @@ def test_config_field_info_happy_admin(client, auth_as, mock_prisma, monkeypatch
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get(
|
||||
"/config/field/info", params={"field_name": "max_parallel_requests"}
|
||||
)
|
||||
response = client.get("/config/field/info", params={"field_name": "max_parallel_requests"})
|
||||
assert response.status_code == 200
|
||||
assert normalize(response.json()) == {
|
||||
"field_name": "max_parallel_requests",
|
||||
|
|
@ -345,9 +341,7 @@ def test_config_field_info_happy_admin(client, auth_as, mock_prisma, monkeypatch
|
|||
}
|
||||
|
||||
|
||||
def test_config_field_info_non_admin_rejected(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_info_non_admin_rejected(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""Non-admin (INTERNAL_USER) is denied — admin-view gate fires."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
|
@ -356,9 +350,7 @@ def test_config_field_info_non_admin_rejected(
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.INTERNAL_USER):
|
||||
response = client.get(
|
||||
"/config/field/info", params={"field_name": "max_parallel_requests"}
|
||||
)
|
||||
response = client.get("/config/field/info", params={"field_name": "max_parallel_requests"})
|
||||
assert response.status_code == 400
|
||||
assert "error" in response.json().get("detail", {})
|
||||
|
||||
|
|
@ -375,16 +367,12 @@ def test_config_field_info_field_not_in_db(client, auth_as, mock_prisma, monkeyp
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get(
|
||||
"/config/field/info", params={"field_name": "max_parallel_requests"}
|
||||
)
|
||||
response = client.get("/config/field/info", params={"field_name": "max_parallel_requests"})
|
||||
assert response.status_code == 400
|
||||
assert "not in DB" in response.json().get("detail", {}).get("error", "")
|
||||
|
||||
|
||||
def test_config_field_info_redacts_nested_secret_for_view_only_admin(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_info_redacts_nested_secret_for_view_only_admin(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""A view-only admin reading a structured field must not receive nested
|
||||
credentials. database_args carries aws_web_identity_token (a DynamoDB
|
||||
role-assumption credential); it must come back redacted while non-secret
|
||||
|
|
@ -405,9 +393,7 @@ def test_config_field_info_redacts_nested_secret_for_view_only_admin(
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY):
|
||||
response = client.get(
|
||||
"/config/field/info", params={"field_name": "database_args"}
|
||||
)
|
||||
response = client.get("/config/field/info", params={"field_name": "database_args"})
|
||||
assert response.status_code == 200
|
||||
value = response.json()["field_value"]
|
||||
assert value["aws_web_identity_token"] == "REDACTED"
|
||||
|
|
@ -415,9 +401,7 @@ def test_config_field_info_redacts_nested_secret_for_view_only_admin(
|
|||
assert value["user_table_name"] == "LiteLLM_UserTable"
|
||||
|
||||
|
||||
def test_config_field_info_full_admin_sees_nested_secret(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_info_full_admin_sees_nested_secret(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""The redaction must not over-redact for a full PROXY_ADMIN, who needs
|
||||
the real nested value to populate the edit form."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
|
|
@ -435,18 +419,14 @@ def test_config_field_info_full_admin_sees_nested_secret(
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get(
|
||||
"/config/field/info", params={"field_name": "database_args"}
|
||||
)
|
||||
response = client.get("/config/field/info", params={"field_name": "database_args"})
|
||||
assert response.status_code == 200
|
||||
value = response.json()["field_value"]
|
||||
assert value["aws_web_identity_token"] == "sk-super-secret-token"
|
||||
assert value["region_name"] == "us-east-1"
|
||||
|
||||
|
||||
def test_config_field_info_redacts_top_level_scalar_for_view_only(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_info_redacts_top_level_scalar_for_view_only(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""The top-level scalar branch must also redact for a view-only admin.
|
||||
database_url carries DB credentials and is not caught by the name masker,
|
||||
so it is in the explicit secret set."""
|
||||
|
|
@ -460,9 +440,7 @@ def test_config_field_info_redacts_top_level_scalar_for_view_only(
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY):
|
||||
response = client.get(
|
||||
"/config/field/info", params={"field_name": "database_url"}
|
||||
)
|
||||
response = client.get("/config/field/info", params={"field_name": "database_url"})
|
||||
assert response.status_code == 200
|
||||
assert response.json()["field_value"] == "REDACTED"
|
||||
|
||||
|
|
@ -476,17 +454,12 @@ def test_redact_general_setting_value_recurses_list_of_dicts():
|
|||
{"path": "/foo", "headers": {"Authorization": "Bearer sk-x"}},
|
||||
{"path": "/bar", "client_secret": "sk-y"},
|
||||
]
|
||||
redacted = ps._redact_general_setting_value(
|
||||
"some_list_field", value, is_full_admin=False
|
||||
)
|
||||
redacted = ps._redact_general_setting_value("some_list_field", value, is_full_admin=False)
|
||||
assert redacted[0]["headers"]["Authorization"] == "REDACTED"
|
||||
assert redacted[0]["path"] == "/foo"
|
||||
assert redacted[1]["client_secret"] == "REDACTED"
|
||||
assert redacted[1]["path"] == "/bar"
|
||||
assert (
|
||||
ps._redact_general_setting_value("some_list_field", value, is_full_admin=True)
|
||||
== value
|
||||
)
|
||||
assert ps._redact_general_setting_value("some_list_field", value, is_full_admin=True) == value
|
||||
|
||||
|
||||
def test_redact_secret_values_in_obj_fails_closed_at_max_depth():
|
||||
|
|
@ -504,22 +477,16 @@ def test_redact_secret_values_in_obj_fails_closed_at_max_depth():
|
|||
for _ in range(ps._REDACT_SECRET_MAX_DEPTH + 2):
|
||||
nested = {"wrap": nested}
|
||||
|
||||
out = ps._redact_general_setting_value(
|
||||
"some_struct_field", nested, is_full_admin=False
|
||||
)
|
||||
out = ps._redact_general_setting_value("some_struct_field", nested, is_full_admin=False)
|
||||
# the secret must not survive anywhere in the returned tree
|
||||
assert "sk-leak-bottom" not in repr(out)
|
||||
|
||||
# full admin is unaffected by the cap — the value comes back untouched
|
||||
admin_out = ps._redact_general_setting_value(
|
||||
"some_struct_field", nested, is_full_admin=True
|
||||
)
|
||||
admin_out = ps._redact_general_setting_value("some_struct_field", nested, is_full_admin=True)
|
||||
assert admin_out is nested
|
||||
|
||||
|
||||
def test_config_list_redacts_pass_through_secret_for_view_only(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_list_redacts_pass_through_secret_for_view_only(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""/config/list must not leak pass_through_endpoints upstream credentials
|
||||
to a view-only admin. pass_through_endpoints is a known secret-bearing
|
||||
field, so a non-admin gets it redacted; a full admin still sees it."""
|
||||
|
|
@ -546,24 +513,16 @@ def test_config_list_redacts_pass_through_secret_for_view_only(
|
|||
)
|
||||
|
||||
def _pass_through_value(body):
|
||||
return next(
|
||||
entry["field_value"]
|
||||
for entry in body
|
||||
if entry["field_name"] == "pass_through_endpoints"
|
||||
)
|
||||
return next(entry["field_value"] for entry in body if entry["field_name"] == "pass_through_endpoints")
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY):
|
||||
view_resp = client.get(
|
||||
"/config/list", params={"config_type": "general_settings"}
|
||||
)
|
||||
view_resp = client.get("/config/list", params={"config_type": "general_settings"})
|
||||
assert view_resp.status_code == 200
|
||||
assert "sk-UPSTREAM-SECRET" not in view_resp.text
|
||||
assert _pass_through_value(view_resp.json()) == "REDACTED"
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
admin_resp = client.get(
|
||||
"/config/list", params={"config_type": "general_settings"}
|
||||
)
|
||||
admin_resp = client.get("/config/list", params={"config_type": "general_settings"})
|
||||
assert admin_resp.status_code == 200
|
||||
admin_value = _pass_through_value(admin_resp.json())
|
||||
assert admin_value[0]["headers"]["Authorization"] == "Bearer sk-UPSTREAM-SECRET"
|
||||
|
|
@ -587,9 +546,7 @@ def test_config_list_happy_admin(client, auth_as, mock_prisma, monkeypatch):
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get(
|
||||
"/config/list", params={"config_type": "general_settings"}
|
||||
)
|
||||
response = client.get("/config/list", params={"config_type": "general_settings"})
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert isinstance(body, list)
|
||||
|
|
@ -695,9 +652,7 @@ def test_config_list_non_admin_rejected(client, auth_as, mock_prisma, monkeypatc
|
|||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
|
||||
with auth_as(LitellmUserRoles.INTERNAL_USER):
|
||||
response = client.get(
|
||||
"/config/list", params={"config_type": "general_settings"}
|
||||
)
|
||||
response = client.get("/config/list", params={"config_type": "general_settings"})
|
||||
assert response.status_code == 400
|
||||
assert "role" in response.json().get("detail", {}).get("error", "").lower()
|
||||
|
||||
|
|
@ -710,9 +665,7 @@ def test_config_list_no_db_error(client, auth_as, monkeypatch):
|
|||
monkeypatch.setattr(ps, "prisma_client", None)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get(
|
||||
"/config/list", params={"config_type": "general_settings"}
|
||||
)
|
||||
response = client.get("/config/list", params={"config_type": "general_settings"})
|
||||
assert response.status_code == 400
|
||||
assert "error" in response.json().get("detail", {})
|
||||
|
||||
|
|
@ -756,9 +709,7 @@ def test_config_field_delete_happy_admin(client, auth_as, mock_prisma, monkeypat
|
|||
}
|
||||
|
||||
|
||||
def test_config_field_delete_non_admin_rejected(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_delete_non_admin_rejected(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""Non-admin caller hits the 400 not-allowed branch with role in detail."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
|
@ -778,9 +729,7 @@ def test_config_field_delete_non_admin_rejected(
|
|||
assert "role" in response.json().get("detail", {}).get("error", "").lower()
|
||||
|
||||
|
||||
def test_config_field_delete_field_not_in_config(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_field_delete_field_not_in_config(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""If there is no general_settings row at all, returns 400 'not in config'."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
|
@ -825,9 +774,7 @@ def test_config_callback_delete_happy_admin(client, auth_as, mock_prisma, monkey
|
|||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.post(
|
||||
"/config/callback/delete", json={"callback_name": "langfuse"}
|
||||
)
|
||||
response = client.post("/config/callback/delete", json={"callback_name": "langfuse"})
|
||||
assert response.status_code == 200
|
||||
# `deleted_at` is an ISO timestamp generated at request time — extend
|
||||
# the volatile set just for this assertion so dict-equality still works.
|
||||
|
|
@ -840,9 +787,7 @@ def test_config_callback_delete_happy_admin(client, auth_as, mock_prisma, monkey
|
|||
}
|
||||
|
||||
|
||||
def test_config_callback_delete_non_admin_rejected(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_config_callback_delete_non_admin_rejected(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""Non-admin caller is rejected with 400 not-allowed."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
|
@ -852,9 +797,7 @@ def test_config_callback_delete_non_admin_rejected(
|
|||
monkeypatch.setattr(ps, "store_model_in_db", True)
|
||||
|
||||
with auth_as(LitellmUserRoles.INTERNAL_USER):
|
||||
response = client.post(
|
||||
"/config/callback/delete", json={"callback_name": "langfuse"}
|
||||
)
|
||||
response = client.post("/config/callback/delete", json={"callback_name": "langfuse"})
|
||||
assert response.status_code == 400
|
||||
assert "role" in response.json().get("detail", {}).get("error", "").lower()
|
||||
|
||||
|
|
@ -869,22 +812,15 @@ def test_config_callback_delete_not_found(client, auth_as, mock_prisma, monkeypa
|
|||
monkeypatch.setattr(ps, "store_model_in_db", True)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={"litellm_settings": {"success_callback": ["slack"]}}
|
||||
)
|
||||
fake_proxy_config.get_config = AsyncMock(return_value={"litellm_settings": {"success_callback": ["slack"]}})
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.post(
|
||||
"/config/callback/delete", json={"callback_name": "langfuse"}
|
||||
)
|
||||
response = client.post("/config/callback/delete", json={"callback_name": "langfuse"})
|
||||
# The handler re-raises HTTPException(404) verbatim (only generic
|
||||
# `Exception` becomes a 500 ProxyException), so pin 404 strictly.
|
||||
assert response.status_code == 404
|
||||
assert (
|
||||
"langfuse" in str(response.json()).lower()
|
||||
or "not found" in str(response.json()).lower()
|
||||
)
|
||||
assert "langfuse" in str(response.json()).lower() or "not found" in str(response.json()).lower()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -948,10 +884,7 @@ def test_get_config_callbacks_internal_error(client, auth_as, mock_prisma, monke
|
|||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
assert response.status_code >= 400
|
||||
assert (
|
||||
"boom" in str(response.json()).lower()
|
||||
or "error" in str(response.json()).lower()
|
||||
)
|
||||
assert "boom" in str(response.json()).lower() or "error" in str(response.json()).lower()
|
||||
|
||||
|
||||
_CALLBACK_ENV_FIXTURE = {
|
||||
|
|
@ -985,14 +918,10 @@ def _install_callbacks_config(monkeypatch, mock_prisma):
|
|||
|
||||
|
||||
def _callback_variables(body: dict, name: str) -> dict:
|
||||
return next(
|
||||
cb["variables"] for cb in body["callbacks"] if cb["name"] == name
|
||||
)
|
||||
return next(cb["variables"] for cb in body["callbacks"] if cb["name"] == name)
|
||||
|
||||
|
||||
def test_get_config_callbacks_redacts_secret_env_vars_for_view_only_admin(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_get_config_callbacks_redacts_secret_env_vars_for_view_only_admin(client, auth_as, mock_prisma, monkeypatch):
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_callbacks_config(monkeypatch, mock_prisma)
|
||||
|
|
@ -1024,9 +953,7 @@ def test_get_config_callbacks_redacts_secret_env_vars_for_view_only_admin(
|
|||
assert otel_vars["OTEL_ENDPOINT"] == _CALLBACK_ENV_FIXTURE["OTEL_ENDPOINT"]
|
||||
|
||||
|
||||
def test_get_config_callbacks_full_admin_still_sees_secret_env_vars(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_get_config_callbacks_full_admin_still_sees_secret_env_vars(client, auth_as, mock_prisma, monkeypatch):
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_callbacks_config(monkeypatch, mock_prisma)
|
||||
|
|
@ -1047,9 +974,7 @@ def test_get_config_callbacks_full_admin_still_sees_secret_env_vars(
|
|||
assert otel_vars["OTEL_HEADERS"] == _CALLBACK_ENV_FIXTURE["OTEL_HEADERS"]
|
||||
|
||||
|
||||
def test_get_config_callbacks_redacts_slack_webhook_urls_for_view_only_admin(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
def test_get_config_callbacks_redacts_slack_webhook_urls_for_view_only_admin(client, auth_as, mock_prisma, monkeypatch):
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
|
|
@ -1170,6 +1095,413 @@ def test_get_config_callbacks_redacts_email_alerting_vars_for_view_only_admin(
|
|||
assert admin_email["SMTP_HOST"] == "smtp.resend.com"
|
||||
|
||||
|
||||
def test_get_config_callbacks_appends_runtime_only_callbacks(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""LIT-5281: a YAML callback that the DB callback list replaced in the merged config still runs, so it must
|
||||
show up as a read_only row next to the editable DB-configured one."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"success_callback": ["langfuse"]},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
import litellm
|
||||
from litellm.integrations.langsmith import LangsmithLogger
|
||||
from litellm.integrations.opentelemetry import OpenTelemetry
|
||||
|
||||
monkeypatch.setattr(litellm, "success_callback", ["langfuse", LangsmithLogger()])
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "callbacks", [OpenTelemetry()])
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
assert response.status_code == 200
|
||||
|
||||
assert [(cb["name"], cb["type"], cb.get("read_only", False)) for cb in response.json()["callbacks"]] == [
|
||||
("langfuse", "success", False),
|
||||
("langsmith", "success", True),
|
||||
("otel", "success_and_failure", True),
|
||||
]
|
||||
|
||||
|
||||
def test_get_config_callbacks_accepts_scalar_and_null_yaml_callbacks(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""`success_callback: langfuse` (a YAML scalar) is one configured callback, not eight single-letter ones, and a
|
||||
`callbacks: null` key contributes nothing."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"success_callback": "langfuse", "failure_callback": None, "callbacks": None},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
import litellm
|
||||
from litellm.integrations.langsmith import LangsmithLogger
|
||||
|
||||
monkeypatch.setattr(litellm, "success_callback", ["langfuse", LangsmithLogger()])
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "callbacks", [])
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
assert response.status_code == 200
|
||||
|
||||
assert [(cb["name"], cb["type"], cb.get("read_only", False)) for cb in response.json()["callbacks"]] == [
|
||||
("langfuse", "success", False),
|
||||
("langsmith", "success", True),
|
||||
]
|
||||
|
||||
|
||||
def test_get_config_callbacks_deduplicates_configured_and_runtime(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""A configured callback shows once as editable, whether the runtime holds its string or an initialized instance
|
||||
(arize initializes an ArizeLogger, logfire a bare OpenTelemetry that only its class identifies)."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"success_callback": ["langfuse", "arize", "logfire"]},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
import litellm
|
||||
from litellm.integrations.arize.arize import ArizeLogger
|
||||
from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig
|
||||
|
||||
arize_logger = ArizeLogger(config=OpenTelemetryConfig(exporter="console"), callback_name="arize")
|
||||
monkeypatch.setattr(litellm, "success_callback", ["langfuse", arize_logger, OpenTelemetry()])
|
||||
monkeypatch.setattr(litellm, "callbacks", [])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [])
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
assert response.status_code == 200
|
||||
|
||||
assert [(cb["name"], cb["type"], cb.get("read_only", False)) for cb in response.json()["callbacks"]] == [
|
||||
("langfuse", "success", False),
|
||||
("arize", "success", False),
|
||||
("logfire", "success", False),
|
||||
]
|
||||
|
||||
|
||||
def test_get_config_callbacks_keeps_yaml_otel_family_callbacks_next_to_configured_one(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
"""LIT-5281: arize, weave_otel and langfuse_otel all initialize OpenTelemetry subclasses. Saving one of them
|
||||
from the dashboard replaces the YAML `callbacks` list, so the YAML siblings keep running and must stay listed
|
||||
under their own names instead of being hidden as duplicates of the configured OTel callback."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"callbacks": ["langfuse_otel"]},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
import litellm
|
||||
from litellm.integrations.arize.arize import ArizeLogger
|
||||
from litellm.integrations.langfuse.langfuse_otel import LangfuseOtelLogger
|
||||
from litellm.integrations.langsmith import LangsmithLogger
|
||||
from litellm.integrations.opentelemetry import OpenTelemetryConfig
|
||||
from litellm.integrations.weave.weave_otel import WeaveOtelLogger
|
||||
|
||||
console_config = OpenTelemetryConfig(exporter="console")
|
||||
monkeypatch.setattr(litellm, "success_callback", [LangsmithLogger()])
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [])
|
||||
monkeypatch.setattr(
|
||||
litellm,
|
||||
"callbacks",
|
||||
[
|
||||
ArizeLogger(config=console_config, callback_name="arize"),
|
||||
WeaveOtelLogger(config=console_config),
|
||||
LangfuseOtelLogger(config=console_config, callback_name="langfuse_otel"),
|
||||
],
|
||||
)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
assert response.status_code == 200
|
||||
|
||||
assert [(cb["name"], cb["type"], cb.get("read_only", False)) for cb in response.json()["callbacks"]] == [
|
||||
("langfuse_otel", "success_and_failure", False),
|
||||
("arize", "success_and_failure", True),
|
||||
("langsmith", "success", True),
|
||||
("weave_otel", "success_and_failure", True),
|
||||
]
|
||||
|
||||
|
||||
def _dotted_path_test_function(*args, **kwargs):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.parametrize("handler_kind", ["instance", "function"])
|
||||
@pytest.mark.parametrize(
|
||||
"config_key,expected_type",
|
||||
[
|
||||
("success_callback", "success"),
|
||||
("failure_callback", "failure"),
|
||||
("callbacks", "success_and_failure"),
|
||||
],
|
||||
)
|
||||
def test_get_config_callbacks_deduplicates_dotted_path_callback(
|
||||
client, auth_as, mock_prisma, monkeypatch, config_key, expected_type, handler_kind
|
||||
):
|
||||
"""A dotted-path callback stays a single editable row instead of duplicating under its class or function name."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
import litellm
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
|
||||
class _DottedPathTestHandler(CustomLogger):
|
||||
pass
|
||||
|
||||
dotted_handler = _DottedPathTestHandler() if handler_kind == "instance" else _dotted_path_test_function
|
||||
dotted_path = f"{__name__}.dotted_handler"
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {config_key: [dotted_path]},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
monkeypatch.setattr(litellm, "callbacks", [dotted_handler])
|
||||
monkeypatch.setattr(litellm, "success_callback", [])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [])
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
|
||||
assert response.status_code == 200
|
||||
callbacks = response.json()["callbacks"]
|
||||
assert [(callback["name"], callback["type"], callback.get("read_only", False)) for callback in callbacks] == [
|
||||
(dotted_path, expected_type, False)
|
||||
]
|
||||
|
||||
|
||||
def test_get_config_callbacks_lists_dict_shaped_config_callbacks(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""Dict-shaped success_callback config values list their keys as editable rows."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"success_callback": {"langsmith": {"batch_size": 1}}},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
import litellm
|
||||
|
||||
monkeypatch.setattr(
|
||||
litellm.logging_callback_manager,
|
||||
"get_callbacks_by_type",
|
||||
MagicMock(return_value={"success": ["langsmith"], "failure": [], "success_and_failure": []}),
|
||||
)
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
|
||||
assert response.status_code == 200
|
||||
callbacks = response.json()["callbacks"]
|
||||
assert [(callback["name"], callback.get("read_only", False)) for callback in callbacks] == [("langsmith", False)]
|
||||
|
||||
|
||||
def test_get_config_callbacks_excludes_internal_runtime_callbacks(client, auth_as, mock_prisma, monkeypatch):
|
||||
"""Proxy infrastructure callbacks are excluded from callback inventory."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"success_callback": []},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
from litellm_enterprise.proxy.hooks.managed_files import _PROXY_LiteLLMManagedFiles
|
||||
|
||||
import litellm
|
||||
from litellm._service_logger import ServiceLogging
|
||||
from litellm.integrations.custom_guardrail import CustomGuardrail
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.integrations.langsmith import LangsmithLogger
|
||||
from litellm.integrations.s3_v2 import S3Logger
|
||||
from litellm.integrations.sqs import SQSLogger
|
||||
from litellm.integrations.vector_store_integrations.vector_store_pre_call_hook import VectorStorePreCallHook
|
||||
from litellm.proxy.hooks.max_budget_limiter import _PROXY_MaxBudgetLimiter
|
||||
from litellm.router import Router
|
||||
|
||||
class _InventoryTestGuardrail(CustomGuardrail):
|
||||
pass
|
||||
|
||||
class _UserCodeLogger(CustomLogger):
|
||||
pass
|
||||
|
||||
def user_code_function(*args, **kwargs):
|
||||
pass
|
||||
|
||||
async def build_aws_loggers() -> tuple[S3Logger, SQSLogger]:
|
||||
return S3Logger(s3_bucket_name="inventory-bucket"), SQSLogger(sqs_queue_url="https://sqs.example/inventory")
|
||||
|
||||
s3_logger, sqs_logger = asyncio.run(build_aws_loggers())
|
||||
router = Router(model_list=[])
|
||||
monkeypatch.setattr(litellm, "input_callback", [])
|
||||
monkeypatch.setattr(
|
||||
litellm, "success_callback", [LangsmithLogger(), s3_logger, router.sync_deployment_callback_on_success]
|
||||
)
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [sqs_logger, router.deployment_callback_on_success])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [user_code_function])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [router.async_deployment_callback_on_failure])
|
||||
monkeypatch.setattr(
|
||||
litellm,
|
||||
"callbacks",
|
||||
[
|
||||
_PROXY_MaxBudgetLimiter(),
|
||||
_PROXY_LiteLLMManagedFiles(internal_usage_cache=MagicMock(), prisma_client=MagicMock()),
|
||||
ServiceLogging(),
|
||||
VectorStorePreCallHook(),
|
||||
_InventoryTestGuardrail(guardrail_name="inventory-test-guardrail"),
|
||||
_UserCodeLogger(),
|
||||
],
|
||||
)
|
||||
monkeypatch.setattr(litellm, "cache", litellm.Cache(type="local"))
|
||||
assert "cache" in litellm.success_callback and "cache" in litellm._async_success_callback
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
response = client.get("/get/config/callbacks")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert [
|
||||
(callback["name"], callback["type"], callback["read_only"]) for callback in response.json()["callbacks"]
|
||||
] == [
|
||||
("_UserCodeLogger", "success_and_failure", True),
|
||||
("langsmith", "success", True),
|
||||
("s3", "success", True),
|
||||
("sqs", "success", True),
|
||||
("user_code_function", "failure", True),
|
||||
]
|
||||
|
||||
|
||||
def test_get_config_callbacks_redacts_runtime_only_row_secrets_for_view_only_admin(
|
||||
client, auth_as, mock_prisma, monkeypatch
|
||||
):
|
||||
"""Runtime-only callback rows are subject to the same redaction gate as configured."""
|
||||
from litellm.proxy import proxy_server as ps
|
||||
from litellm.proxy._types import LitellmUserRoles
|
||||
|
||||
_install_litellm_config(mock_prisma)
|
||||
monkeypatch.setattr(ps, "prisma_client", mock_prisma)
|
||||
monkeypatch.setattr(ps, "llm_router", None)
|
||||
|
||||
fake_proxy_config = MagicMock()
|
||||
fake_proxy_config.get_config = AsyncMock(
|
||||
return_value={
|
||||
"litellm_settings": {"success_callback": []},
|
||||
"general_settings": {},
|
||||
"environment_variables": dict(_CALLBACK_ENV_FIXTURE),
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(ps, "proxy_config", fake_proxy_config)
|
||||
|
||||
import litellm
|
||||
|
||||
monkeypatch.setattr(litellm, "success_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_success_callback", [])
|
||||
monkeypatch.setattr(litellm, "failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "_async_failure_callback", [])
|
||||
monkeypatch.setattr(litellm, "callbacks", ["otel"])
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY):
|
||||
response = client.get("/get/config/callbacks")
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
|
||||
callbacks = body["callbacks"]
|
||||
otel_cb = next((cb for cb in callbacks if cb["name"] == "otel"), None)
|
||||
assert otel_cb is not None
|
||||
assert otel_cb["type"] == "success_and_failure"
|
||||
assert otel_cb["read_only"] is True
|
||||
assert otel_cb["variables"]["OTEL_HEADERS"] == "REDACTED"
|
||||
assert otel_cb["variables"]["OTEL_ENDPOINT"] == _CALLBACK_ENV_FIXTURE["OTEL_ENDPOINT"]
|
||||
|
||||
with auth_as(LitellmUserRoles.PROXY_ADMIN):
|
||||
admin_response = client.get("/get/config/callbacks")
|
||||
assert admin_response.status_code == 200
|
||||
admin_body = admin_response.json()
|
||||
admin_otel = next((cb for cb in admin_body["callbacks"] if cb["name"] == "otel"), None)
|
||||
assert admin_otel is not None
|
||||
assert admin_otel["variables"]["OTEL_HEADERS"] == _CALLBACK_ENV_FIXTURE["OTEL_HEADERS"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET /config/yaml
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1183,9 +1515,7 @@ def test_config_yaml_returns_demo_payload(client, auth_as):
|
|||
response = client.request("GET", "/config/yaml", json={})
|
||||
shape = {
|
||||
"status": response.status_code,
|
||||
"media_type_yaml": response.headers.get("content-type", "").startswith(
|
||||
"application/json"
|
||||
),
|
||||
"media_type_yaml": response.headers.get("content-type", "").startswith("application/json"),
|
||||
"has_body": len(response.content) > 0,
|
||||
}
|
||||
assert shape == {
|
||||
|
|
|
|||
|
|
@ -97,6 +97,24 @@ describe("LoggingCallbacksTable", () => {
|
|||
expect(onDelete).toHaveBeenCalledWith(callback);
|
||||
});
|
||||
|
||||
it("shows a read-only label instead of the actions menu for runtime-only callback rows", () => {
|
||||
render(
|
||||
<LoggingCallbacksTable
|
||||
callbacks={[
|
||||
{ name: "langfuse", type: "success" as const, variables: baseVars },
|
||||
{ name: "datadog", type: "success" as const, variables: baseVars, read_only: true },
|
||||
]}
|
||||
availableCallbacks={{}}
|
||||
onTest={vi.fn()}
|
||||
onEdit={vi.fn()}
|
||||
onDelete={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
expect(screen.getByTestId("callback-actions-langfuse-success")).toBeInTheDocument();
|
||||
expect(screen.queryByTestId("callback-actions-datadog-success")).not.toBeInTheDocument();
|
||||
expect(screen.getAllByText("Read only")).toHaveLength(1);
|
||||
});
|
||||
|
||||
// Regression: `/get_callbacks` returns the same `name` twice when a
|
||||
// callback is registered for both success and failure (e.g. `generic_api`
|
||||
// → POST to spend-log on both 200 and 4xx/5xx). The UI used to ignore
|
||||
|
|
|
|||
|
|
@ -50,6 +50,16 @@ interface CallbackRowActionsProps {
|
|||
}
|
||||
|
||||
function CallbackRowActions({ callback, onTest, onEdit, onDelete }: CallbackRowActionsProps) {
|
||||
if (callback.read_only) {
|
||||
return (
|
||||
<span
|
||||
className="text-xs text-muted-foreground"
|
||||
title="Active callback that was not added through the dashboard. Edit it where it was configured."
|
||||
>
|
||||
Read only
|
||||
</span>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ export interface AlertingObject {
|
|||
// every row to render as "Success".
|
||||
type?: "success" | "failure" | "success_and_failure";
|
||||
variables: AlertingVariables;
|
||||
read_only?: boolean;
|
||||
}
|
||||
|
||||
export interface AlertingVariables {
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue