* fix(vector-stores): keep proxy context out of provider vector store metadata
The proxy merged its request context (user_api_key_* fields, endpoint,
user_agent, requester_ip_address, litellm_api_version) and the router's
deployment info into the vector store request's metadata. For vector
stores that field is also OpenAI's own metadata, which is capped at 16
keys, so a store created with a team key was full on creation and every
later metadata update returned 400.
Vector store routes now keep that context in litellm_metadata, like
batches, files and responses, and the router does the same for
avector_store_create. Provider-side metadata holds only what the caller
sent.
Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in
script that removes the proxy-written keys from stores created before
this change. It is a dry run unless --apply is passed.
* fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values
Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values.
* fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata
Vector store routes now keep proxy context in litellm_metadata, but the v3
limiter only read agent_id from metadata, and logging let provider metadata
tags or spend_logs_metadata replace the proxy's values. A string tags value
also crashed get_request_tags.
The limiter now prefers the proxy-resolved agent_id in litellm_metadata.
Logging combines list tags from both buckets (metadata first, deduplicated)
and merges dict spend_logs_metadata with litellm_metadata winning, ignoring
values of the wrong type.
* fix(vector-stores): narrow the logging fallback to mistyped caller values
The tag union and spend_logs_metadata merge changed logged tags and
spend_logs_metadata on every route that has both metadata buckets, deduped
tags quadratically, and stopped background interaction polling from keeping
string tags as sent. Caller values now win exactly as before, and the
proxy's litellm_metadata tags or spend_logs_metadata only replace a caller
value of the wrong type.
The limiter reads agent ids through a pydantic model instead of a cast and
ignores non-string ids.
* fix(vector-stores): log the proxy's full tag list on litellm_metadata routes
On routes that keep proxy context in litellm_metadata, the proxy builds
litellm_metadata.tags from the caller's tags plus key, team and header
tags, but get_request_tags preferred the caller's metadata.tags. A caller
sending metadata.tags on a vector store call dropped the key's tags from
request_tags, which tag spend tracking reads. get_request_tags now prefers
a non-empty litellm_metadata tag list, and falls back to metadata.tags as
before.
* fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets
Moving vector store proxy context into litellm_metadata broke shared readers
(agent limits, request tags, merged logging metadata), and fixing them
widened the change to every litellm_metadata route. This reverts the bucket
move, the router change and the reader fixes, so proxy context stays in
metadata exactly as on main.
Vector store create and update now drop the keys LiteLLM itself writes
(user_api_key*, requester and deployment keys, agent ids) from the metadata
sent to the provider, in the shared optional-param helper both paths use.
The cleanup script reuses the same key list.
* fix(vector-stores): pass non-string metadata keys through the provider metadata filter
* fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls
* fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy
* fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies
* fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed
* fix(ui): explain unbackfilled key lifetime spend and ship a backfill script
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): cover linked budgets, deleted keys and double-hashed logs in total_spend backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): count duplicate archived tokens once in total_spend backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): only rebuild resetting archived rows in total_spend backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): split spend log rebuild into opt-in backfill_key_total_spend_from_spend_logs.sql
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(db_scripts): scope backfill verify query to non-resetting keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): lift every key to at least current spend in backfill_key_total_spend.sql
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(db_scripts): align spend log backfill header with lifted floor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): point resetting keys at the spend log backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): drop script name from lifetime spend tooltip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
partition_spend_logs.sql and unpartition_spend_logs.sql hardcode every
index Prisma defines on LiteLLM_SpendLogs, because `LIKE ... INCLUDING
DEFAULTS INCLUDING GENERATED` copies columns but not indexes. They also
rename the old table's indexes aside first, since index names are unique
per schema and a surviving name makes `CREATE INDEX IF NOT EXISTS` a
silent no-op.
The new (api_key, startTime) index was in neither list, so an operator who
partitions (or unpartitions) after this migration lands gets a replacement
table without it, and Prisma will not recreate it: the migration is already
recorded as applied, and `migrate deploy` skips its drift sanity check when
there is nothing pending.
Verified on Postgres 18 against the shipped migration statements:
* unpatched partition script -> parent table has no api_key index, and
re-running the migration's own `CREATE INDEX IF NOT EXISTS` reports
success while being skipped, because the legacy table still owns the
name. The recovery an operator would reach for silently does nothing.
* patched -> index survives partitioning, propagates to every partition
(LiteLLM_SpendLogs_p*_api_key_startTime_idx) and to DEFAULT, is chosen
by the planner for the key+date-range query shape the endpoints use,
and survives the unpartition round-trip with all rows intact.
The backfill compares the naive start_time column against a timestamptz
cutover, and that coercion follows the session time zone, so a non-UTC
session shifts the cutover boundary by the offset. Pinning the session
makes the whole script timezone-independent. The date bucketing itself was
already safe: to_char on a timestamp without time zone ignores the session
time zone and the stored values are UTC
GET /v1/tool/spend served the Cost Optimization card with two raw queries
over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load;
the totals query's driving scan was all of SpendLogs in the window. Both
per-request tables reach 1M+ rows at customer scale, so the card cost
O(traffic) per view and had to be capped at 30 days.
The index writer also mined proxy_server_request.tools, i.e. tools DECLARED
in the request body, attributing each request's full spend to tools that
never ran; and all non-MCP mining ran against payload fields that are '{}'
unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently
depended on a privacy setting.
Now the spend writer builds a ToolUsageTransaction at request time from
invoked tools only, resolved by the shared get_tool_calls_from_response
normalizer so every response surface (chat completions, Responses API,
Anthropic Messages) is covered; the tool registry's response arm delegates
to the same owner. Transactions queue beside the spend-log queue and the
flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup
(date, tool_name PK) in one transaction, retrying connection errors with
backoff (a failed batch commits nothing, so the retry cannot double-count)
and dropping the batch with an error log on anything else.
The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS
tools by spend via group_by and daily covers only those tools, so the
response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or
tool-name cardinality; the 30-day clamp is gone. total_spend is dropped
from the response; it was never rendered and its deduplicated semantics
are not computable from a rollup. Spend-log retention deliberately does
not touch the rollup, so tool spend history outlives per-request rows.
Drops PLR0915 from ruff's extend-select along with its per-file-ignores,
and strips the now-unused `# noqa: PLR0915` directives across the codebase
(RUF100 would otherwise flag them as unused). The C901 suppression that
shared a directive with PLR0915 in streaming_handler.py is preserved.
High-volume deployments see LiteLLM_SpendLogs grow unbounded because
retention via DELETE leaves dead tuples that autovacuum cannot reclaim
fast enough. With a range-partitioned table, retention drops whole
partitions instead: an instant metadata operation that returns disk to
the OS immediately.
The feature is gated behind general_settings.use_spend_logs_partitioning
(default false). With the flag off, the cleanup job never queries the
catalog and behaves exactly as today. With it on, the job verifies the
table is partitioned, pre-creates upcoming partitions, and drops expired
ones; expired rows the drops cannot reach (DEFAULT partition, partitions
spanning the cutoff) are still deleted row-wise so retention is never
bypassed. If the table is not partitioned it falls back to batched
DELETE only.
Converting an existing table is a manual, documented operation in
db_scripts/partition_spend_logs.sql; db_scripts/unpartition_spend_logs.sql
rolls it back. Both scripts rename the old table's indexes aside before
recreating them, since a table rename keeps the schema-unique index names
and would otherwise silently skip the CREATE INDEX IF NOT EXISTS block.
Granularity and pre-create lookahead are tunable via
SPEND_LOG_PARTITION_INTERVAL (day/week/month, invalid values fall back to
day) and SPEND_LOG_PARTITION_PRECREATE_AHEAD.
* Add O3-Mini for Azure and Remove Vision Support (#8161)
* Azure Released O3-mini at the same time as OAI, so i've added support here. Confirmed to work with Sweden Central.
* [FIX] replace cgi for python 3.13 with email.Message as suggested in PEP 594 (#8160)
* Update model_prices_and_context_window.json (#8120)
codestral2501 pricing on vertex_ai
* Fix/db view names (#8119)
* Fix to case sensitive DB Views name
* Fix to case sensitive DB View names
* Added quotes to check query as well
* Added quotes to create view query
* test: handle server error for flaky test
vertex ai has unstable endpoints
---------
Co-authored-by: Wanis Elabbar <70503629+elabbarw@users.noreply.github.com>
Co-authored-by: Honghua Dong <dhh1995@163.com>
Co-authored-by: superpoussin22 <vincent.nadal@orange.fr>
Co-authored-by: Miguel Armenta <37154380+ma-armenta@users.noreply.github.com>
* fix(model_prices_and_context_window.json): add cost tracking for more vertex llama3.1 model
8b and 70b models
* fix(proxy/utils.py): handle data being none on pre-call hooks
* fix(proxy/): create views on initial proxy startup
fixes base case, where user starts proxy for first time
Fixes https://github.com/BerriAI/litellm/issues/5756
* build(config.yml): fix vertex version for test
* feat(ui/): support enabling/disabling slack alerting
Allows admin to turn on/off slack alerting through ui
* feat(rerank/main.py): support langfuse logging
* fix(proxy/utils.py): fix linting errors
* fix(langfuse.py): log clean metadata
* test(tests): replace deprecated openai model
* fix(proxy/utils.py): comment out auto-upsert logic in check_view_exists
Prevents proxy from failing on startup due to faulty logic
* fix(db/migration_scripts/create_views.py): fix 'DailyTagSpend' quotation on check
* fix(create_views.py): mongly global spend time period should be 30d not 20d
* fix(schema.prisma): index on startTime and endUser for efficient UI querying
