litellm/litellm/vector_stores/utils.py
yuneng-jiang 4846cfabc7
fix(vector-stores): keep proxy context out of provider vector store metadata (#47021)
* fix(vector-stores): keep proxy context out of provider vector store metadata

The proxy merged its request context (user_api_key_* fields, endpoint,
user_agent, requester_ip_address, litellm_api_version) and the router's
deployment info into the vector store request's metadata. For vector
stores that field is also OpenAI's own metadata, which is capped at 16
keys, so a store created with a team key was full on creation and every
later metadata update returned 400.

Vector store routes now keep that context in litellm_metadata, like
batches, files and responses, and the router does the same for
avector_store_create. Provider-side metadata holds only what the caller
sent.

Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in
script that removes the proxy-written keys from stores created before
this change. It is a dry run unless --apply is passed.

* fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values

Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values.

* fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata

Vector store routes now keep proxy context in litellm_metadata, but the v3
limiter only read agent_id from metadata, and logging let provider metadata
tags or spend_logs_metadata replace the proxy's values. A string tags value
also crashed get_request_tags.

The limiter now prefers the proxy-resolved agent_id in litellm_metadata.
Logging combines list tags from both buckets (metadata first, deduplicated)
and merges dict spend_logs_metadata with litellm_metadata winning, ignoring
values of the wrong type.

* fix(vector-stores): narrow the logging fallback to mistyped caller values

The tag union and spend_logs_metadata merge changed logged tags and
spend_logs_metadata on every route that has both metadata buckets, deduped
tags quadratically, and stopped background interaction polling from keeping
string tags as sent. Caller values now win exactly as before, and the
proxy's litellm_metadata tags or spend_logs_metadata only replace a caller
value of the wrong type.

The limiter reads agent ids through a pydantic model instead of a cast and
ignores non-string ids.

* fix(vector-stores): log the proxy's full tag list on litellm_metadata routes

On routes that keep proxy context in litellm_metadata, the proxy builds
litellm_metadata.tags from the caller's tags plus key, team and header
tags, but get_request_tags preferred the caller's metadata.tags. A caller
sending metadata.tags on a vector store call dropped the key's tags from
request_tags, which tag spend tracking reads. get_request_tags now prefers
a non-empty litellm_metadata tag list, and falls back to metadata.tags as
before.

* fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets

Moving vector store proxy context into litellm_metadata broke shared readers
(agent limits, request tags, merged logging metadata), and fixing them
widened the change to every litellm_metadata route. This reverts the bucket
move, the router change and the reader fixes, so proxy context stays in
metadata exactly as on main.

Vector store create and update now drop the keys LiteLLM itself writes
(user_api_key*, requester and deployment keys, agent ids) from the metadata
sent to the provider, in the shared optional-param helper both paths use.
The cleanup script reuses the same key list.

* fix(vector-stores): pass non-string metadata keys through the provider metadata filter

* fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls

* fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy

* fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies

* fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed
2026-10-17 12:51:47 -07:00

139 lines
5.1 KiB
Python

import copy
from collections.abc import Container, Mapping
from typing import Final, cast, get_type_hints
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
from litellm.constants import ROUTING_REQUEST_TAGS_METADATA_KEY, VECTOR_STORE_CALLER_METADATA_KWARG
from litellm.llms.base_llm.vector_store.transformation import BaseVectorStoreConfig
from litellm.types.vector_stores import (
VectorStoreCreateOptionalRequestParams,
VectorStoreSearchOptionalRequestParams,
)
LITELLM_WRITTEN_METADATA_KEYS: Final = frozenset(
{
"requester_ip_address",
"user_agent",
"endpoint",
"litellm_api_version",
"deployment",
"deployment_model_name",
"api_base",
"agent_id",
"actor_agent_id",
"target_agent_id",
"billing_agent_id",
"agent_execution_mode",
"verified_human_user_id",
"litellm_caller_session_id",
}
)
def is_litellm_written_metadata_key(key: str) -> bool:
return key == "user_api_key" or key.startswith("user_api_key_") or key in LITELLM_WRITTEN_METADATA_KEYS
def _is_callers_key(key: object) -> bool:
return not isinstance(key, str) or not is_litellm_written_metadata_key(key)
def _sent_by_proxy_or_router(params: Mapping[str, object], metadata: Container[object]) -> bool:
return params.get("proxy_server_request") is not None or ROUTING_REQUEST_TAGS_METADATA_KEY in metadata
def _callers_metadata(metadata: Mapping[object, object]) -> Mapping[object, object]:
return {key: value for key, value in metadata.items() if _is_callers_key(key)}
_JSON_OBJECT: Final = TypeAdapter(Mapping[str, object])
class VectorStoreWriteBody(BaseModel):
model_config = ConfigDict(frozen=True, extra="ignore")
metadata: object = None
def _json_object(raw: str | bytes) -> Mapping[str, object] | None:
try:
return _JSON_OBJECT.validate_json(raw)
except ValidationError:
return None
def _decoded_metadata(metadata: object) -> object:
if not isinstance(metadata, str):
return metadata
decoded: Final = _json_object(metadata)
return metadata if decoded is None else decoded
def callers_metadata_snapshot(raw_json_body: bytes | None, parsed_body: VectorStoreWriteBody) -> object:
raw_body: Final = None if raw_json_body is None else _json_object(raw_json_body)
metadata: Final = copy.deepcopy(parsed_body.metadata) if raw_body is None else raw_body.get("metadata")
return _decoded_metadata(metadata)
def _metadata_for_provider(params: Mapping[str, object]) -> object:
if VECTOR_STORE_CALLER_METADATA_KWARG in params:
return params[VECTOR_STORE_CALLER_METADATA_KWARG]
metadata: Final = params.get("metadata")
if not isinstance(metadata, Mapping):
return metadata
request_metadata: Final = cast(Mapping[object, object], metadata) # cast-ok: request metadata is a JSON object
if not _sent_by_proxy_or_router(params, request_metadata):
return request_metadata
return _callers_metadata(request_metadata)
class VectorStoreRequestUtils:
"""Helper utils for constructing Vector Store search requests"""
@staticmethod
def get_requested_vector_store_search_optional_param(
params: dict[str, object],
vector_store_provider_config: BaseVectorStoreConfig,
) -> VectorStoreSearchOptionalRequestParams:
"""
Filter parameters to only include those defined in VectorStoreSearchOptionalRequestParams.
Args:
params: Dictionary of parameters to filter
Returns:
VectorStoreSearchOptionalRequestParams instance with only the valid parameters
"""
valid_keys: Final = get_type_hints(VectorStoreSearchOptionalRequestParams).keys()
filtered_params: Final = {k: v for k, v in params.items() if k in valid_keys and v is not None}
optional_params: Final = vector_store_provider_config.map_openai_params(
non_default_params=params,
optional_params=filtered_params,
drop_params=False,
)
return cast(VectorStoreSearchOptionalRequestParams, optional_params)
@staticmethod
def get_requested_vector_store_create_optional_param(
params: dict[str, object],
) -> VectorStoreCreateOptionalRequestParams:
"""
Filter parameters to only include those defined in VectorStoreCreateOptionalRequestParams.
Args:
params: Dictionary of parameters to filter
Returns:
VectorStoreCreateOptionalRequestParams instance with only the valid parameters
"""
valid_keys: Final = get_type_hints(VectorStoreCreateOptionalRequestParams).keys() - {"metadata"}
filtered_params: Final = {k: v for k, v in params.items() if k in valid_keys and v is not None}
metadata: Final = _metadata_for_provider(params)
if metadata is None:
return cast(VectorStoreCreateOptionalRequestParams, filtered_params)
return cast( # cast-ok: same params as above plus the metadata the provider should store
VectorStoreCreateOptionalRequestParams, {**filtered_params, "metadata": metadata}
)