mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
19605 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
65ef103ddf |
feat(ui): add TypeSafe and Strands Decider to the Add Model provider list (#45335)
* feat(ui): add TypeSafe and Strands Decider to the Add Model provider list
* test(public_endpoints): type the provider fields helper against the route's model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
3e78f8d25d |
test: realign stale decisions and lens tests with merged behaviour (#45346)
Co-authored-by: yuneng <yuneng@berri.ai>
(cherry picked from commit
|
||
|
yuneng-jiang
|
42ce1ab40a |
test(decisions): post System One bodies to /v1/systemone in the translation bases (#45327)
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
806480c2ff |
refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler (#45130)
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): keep APIConnectionError for unreachable deployments on every provider
`_map_upstream_exception` now maps a status `_handle_error` synthesized for a
non-HTTP failure to `APIConnectionError` before `exception_type` runs, so a
Perplexity (openai-compatible) deployment that cannot be reached no longer
answers `InternalServerError` where main answers `APIConnectionError`.
Audit cells: the unreachable-deployment integration cell is parametrized over
the five providers, OpenRouter connection failures are pinned on chat, stream,
embeddings, decisions and the SDK, a deployment whose provider has no decisions
config gets the 400 naming every supported provider, and a unit cell pins the
mapper alone. The first docstring sentence of the five translation bases is
dropped.
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
1b4807d4f0 |
feat(decisions): add the OpenAI Decisions spec types and the System One translation (#45129)
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): match the SDK on strict choice values and drop the invented min_length bounds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): compose DecisionsRequest from model and body and read the translator top down
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): use the plural Decisions prefix only for the request and response envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): import assert_never from typing_extensions for Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit
|
||
|
yucheng-berri
|
c00f2b8134 |
feat(guardrails): backport the decision model guardrail to stable/1.104.x (#45663)
Cherry-pick of the rc/1.105.0 backport of #45663 (#45894), which already adapts #45663 to branches that predate litellm/decisions/call.py, GuardrailReadOnlyDetails.tsx, LiteLLMBaseModel, @/lib/decisionModels and the TypeSafe Add Model provider. stable/1.104.x has the same decisions providers as rc/1.105.0. - The decision model unit test lives in tests/test_litellm/proxy/guardrails/ guardrail_hooks, where stable/1.104.x keeps the guardrail hook tests - The OpenAPI snapshot and schema.d.ts match a regeneration on this branch (cherry picked from commit |
||
|
mateo-berri
|
7446e02690 |
fix(decisions): return guardrail_information on OpenAI-format decisions when requested
Backport of the #45184 and #45214 guardrail fix. Cherry-picked from |
||
|
mateo-berri
|
f0cb1d5757 | fix(decisions): send OpenAI instructions for System One questions without them and reject single-option questions | ||
|
mateo-berri
|
5d2cc109c8 |
test(decisions): clear every OpenAI base setting in the proxy OpenAI deployment test
Backport of #45214 to stable/1.104.x.
Cherry-picked from
|
||
|
mateo-berri
|
c65d369958 |
fix(decisions): honor litellm OpenAI key and base settings for the openai provider
Backport of #45214 to stable/1.104.x.
Cherry-picked from
|
||
|
mateo-berri
|
7df9c0620f |
fix(decisions): load on Python 3.10 and 3.11 and bill cached tokens once under custom pricing
Backport of
|
||
|
mateo-berri
|
2976595063 |
fix(decisions): keep provider fields inside System One answers
Backport of #45214 to stable/1.104.x. Cherry-picked from |
||
|
mateo-berri
|
df8f46c4ea |
test(decisions): read _hidden_params in the OpenAI provider test on stable/1.104.x
The public hidden_params accessor comes from #44668, which is not on stable/1.104.x, so the backported test reads the private attribute like the rest of the file |
||
|
mateo-berri
|
637cc3f5b1 |
feat(decisions): route both request formats through a shared decisions IR
System One and OpenAI-format bodies now convert to one internal representation, and each provider translates it to its own wire format. OpenAI deployments receive the caller's messages, images, names, typed choice values, level descriptions and safety_identifier unchanged. System One providers return a 400 for image input. Cached and cache-write tokens are priced on both response shapes, and provider response extras survive the round trip.
(cherry picked from commit
|
||
|
mateo-berri
|
fbcc73b579 |
refactor(decisions): move the OpenAI wire translation into llms/openai
(cherry picked from commit
|
||
|
mateo-berri
|
138f901e09 |
feat(decisions): add OpenAI as a Decisions provider
System One requests to openai/ models are translated to OpenAI's Decisions shape on the way out and OpenAI's answers are translated back, so both /v1/systemone and /v1/decisions can route to gpt-6-luna.
(cherry picked from commit
|
||
|
mateo-berri
|
d9afde2a63 |
fix(decisions): return 400 for bodies that are not JSON
Backport of the #45184 follow-up to stable/1.104.x.
Cherry-picked from
|
||
|
mateo-berri
|
5f4588dec4 |
fix(decisions): cap OpenAI-format questions at the System One limit
Backport of the #45184 follow-up to stable/1.104.x.
Cherry-picked from
|
||
|
mateo-berri
|
c24cbf6442 |
feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions
Backport of #45184 to stable/1.104.x.
Cherry-picked from
|
||
|
devin-ai-integration[bot]
|
bc4b7fd42c |
feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236)
Backport of #44236 to stable/1.104.x.
Cherry-picked from
|
||
|
Yuneng Jiang
|
5bb3b88197
|
fix(proxy-extras): keep the post-migration sanity check from building the hand-built SpendLogs indexes
schema.prisma still declares the LiteLLM_SpendLogs (api_key, startTime) and (litellm_call_id) indexes while their migrations are now SELECT 1, so the v1 resolver's drift script recreated both with a blocking CREATE INDEX at boot. The drift script now drops those two statements before it is applied, matching filter_request_log_index_diff on main. The schema sync test exempts the two indexes
Ported to rc/1.104.0 from stable/1.103.x (#44286), which is not on main: main carries filter_request_log_index_diff from #43948 instead, and its backport here was reverted in #44212. The key metadata recovery hunk is dropped because this line's test file has no Postgres-backed index fixture
(cherry picked from commit
|
||
|
yuneng-jiang
|
5aa0cd00fb
|
chore(release): backport part of #43902 and #43983 to rc/1.104.0 (#44332)
* test(e2e): align completion, SAIL, and spend-log fixtures with supported contracts (#43902) Partial backport to rc/1.104.0: only the text completion endpoint test moves off the deprecated gpt-3.5-turbo-instruct to openai/gpt-5.4-nano. The SAIL, spend-log and coverage registry repairs target tests or behavior this line does not need for the e2e suite (cherry picked from commit |
||
|
yuneng-jiang
|
31d6c60e92
|
chore(release): backport #44283 and part of #44229 to rc/1.104.0 (#44312)
* test: repair stale and polluting tests red on scheduled main CI (#44229) Partial backport to rc/1.104.0: only the owned Redis port, Presidio stored-content check and event-loop lag logging fixes apply here. The other repairs target tests or behavior this line does not have * test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context * test(secret-detection): give the hand-built redaction request an ASGI path Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path * test(integration): isolate litellm callback lists per sdk test usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards * test(integration): keep the owner-lookup fault proxy off the shared read replica The owned proxy points DATABASE_URL at a scratch database but inherited DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared database and rejected the freshly created key with token_not_found_in_db. Drop the replica variable like the other scratch-database owned proxies * test(integration): request every seeded key in the team owner breakdown The aggregated team activity endpoint now caps breakdown.api_keys at the top 100 keys by default (#43398), so the 300 seeded keys came back as 100 rows. The test guarantees each key is reported with its own owner, so ask for an api_key_limit that covers all seeded keys * test(integration): give every owned Redis its own port in the redis-cache container On CircleCI every owned Redis ran on the fixed port 16379 inside the shared redis-cache container. When an earlier server still held that port, the new one failed to bind, readiness pinged the old server, the pidfile read failed and cleanup then reported "Owned Redis still serves after shutdown" Reserve an ephemeral port for the docker-exec path the same way the local binary path already does, and refuse to start when something already serves the chosen port so the failure names the real cause * test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail * test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to * test(e2e): check only stored message content for a leaked card number The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there * test(e2e): assert the proxy decodes token-array embeddings for titan The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer * test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5 * test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed * test(e2e): cite the tokenizer and date behind the titan token-array fixture * test(e2e): let migration seed replicas finish their request-log indexes before cloning Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema * test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope * test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly (cherry picked from commit |
||
|
yuneng-jiang
|
656daa062d
|
chore(release): backport #44277 to rc/1.104.0 and bump oauthlib (#44304)
* fix(proxy): stop queued registry read-throughs spending the resync budget (#44277)
* fix(proxy): stop queued registry read-throughs spending the resync budget
RegistryReadThrough.attempt serializes misses behind one lock, but every
request that was queued behind the first one still spent a unit of the
20-per-5s resync budget and re-ran the DB resync, even though the first
request had already loaded the object. A burst of more than 20 requests
for a model created on another worker therefore exhausted the budget and
the rest got 400 "Invalid model name".
attempt now checks whether the key is already loaded once it holds the
lock and returns early without touching the budget. Models check the
router's model names and deployment ids, guardrails and agents reuse
their existing registry lookups.
* test(proxy): gate the queued read-through test on events and cover each registry's loaded check
The queued-requests test now holds the first resync on an asyncio.Event instead of
a timed sleep and records calls in a recorder with tuple and frozenset state. New
tests show the model, guardrail and agent read-throughs each answer an object that
is already loaded without reading the database, so rewiring any registry's loaded
check now fails a test
* test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture
(cherry picked from commit
|
||
|
yuneng-jiang
|
a1f27fcbe1
|
test: fix three order-dependent and timing-flaky tests (#44271) (#44280)
* test(integration): answer the model-info refresh GET in the mixed MCP responses wire peer
The proxy's periodic model-info refresh sends GET /v1/models to the deployment api_base, which tripped the peer's /responses-only assertion when a tick landed mid-test
* test(e2e): wait for the api-keys URL after clicking Virtual Keys in onboarding
/ui already renders the Virtual Keys heading, so the helper returned before navigation finished. The late route change moved focus and closed the account menu popover in hideLiteAdmin
* test(proxy): stop two unit modules leaking app.openapi_schema and a session-wide Router
test_custom_openapi cached a stripped schema on app.openapi_schema and never cleared it, breaking later openapi route tests. test_proxy_reject_logging built a module-level Router that stayed in the live router registry all session and re-added cost-map keys during a reload. Reset the schema via monkeypatch and make the Router a function-scoped fixture
(cherry picked from commit
|
||
|
yuneng-jiang
|
6e747b059b
|
test(integration): opt the config pass-through spend-log case into auth (#44267)
The failure spend-log row from #42695 is written for pass-through routes that
run as LLM API routes, which a config route only does with auth: true. The
test omitted auth and passed only while config wins (#41779) registered config
entries through the typed model, where auth defaults to true. #43962 restored
the pre-config-wins registration, so the route lost that status and the row
was never written. Set auth: true on the route so the test covers the logging
it was written for without depending on that side effect
(cherry picked from commit
|
||
|
yuneng-jiang
|
e0127558f2
|
Merge pull request #44206 from BerriAI/litellm_rc_1_104_0_enforce_migration_check_default
fix(proxy): enforce the migration check by default and stop building SpendLogs indexes in migrations (rc/1.104.0) |
||
|
yuneng-jiang
|
fb90b661a2
|
chore(release): backport #44066 to rc/1.104.0 (#44216)
* feat(mcp)!: disable stdio MCP servers by default (#44066)
* feat(mcp)!: disable stdio MCP servers by default
stdio MCP servers now only run when the proxy is started with
LITELLM_ENABLE_MCP_STDIO=true. While it is off, existing stdio servers stay
registered but never start: tool listings skip them quietly, direct tool
calls and health checks return a 403 naming the env var, and creating or
updating a stdio server is rejected. The flag is read from the process
environment only, so DB-stored environment_variables cannot turn it on.
The UI reads mcp_stdio_enabled from /.well-known/litellm-ui-config to grey
out the stdio transport, show a banner on stdio forms, and badge stdio
server cards.
BREAKING CHANGE: stdio MCP servers are off by default. Set
LITELLM_ENABLE_MCP_STDIO=true in the proxy environment and restart to keep
using them.
* fix(mcp): ignore stdio flag from config file and read UI flag from the selected worker
LITELLM_ENABLE_MCP_STDIO set under environment_variables in config.yaml is now skipped like the DB-stored value, so only the process environment can enable stdio. The dashboard reads mcp_stdio_enabled from the proxy it is managing, so a control plane shows each worker's own setting.
* test(mcp): cover non-mapping payloads in the shared transport validator
* fix(mcp): skip blocked stdio servers quietly in every listing and keep the UI unchanged until the flag loads
Prompt, resource and resource-template listings now skip a blocked stdio server at debug level like tool listing does, instead of logging a warning per server on every call. The dashboard only treats stdio as disabled once the proxy explicitly reports mcp_stdio_enabled false, so a proxy with the flag on, or an older one without the field, renders exactly as before with no flicker while loading.
* fix(mcp): route blocked stdio tool calls to the flag error and warn once per server
A gateway tools/call naming a blocked stdio server's tool now returns the
LITELLM_ENABLE_MCP_STDIO message instead of "Tool not found".
The "will not start" warning moves out of build_mcp_server_from_table, which
DB reload re-runs on every cycle for rows with a NULL updated_at and which
drafts and test-connection also call. It now fires when a row first enters
the registry or changes transport.
* fix(ui): explain on the server detail page why a stdio server is inert
The Overview and MCP Tools tabs showed "No tools available" with no reason
while stdio is disabled. The detail page now shows the same warning banner
as the edit form, and hands off to the form's banner once editing starts.
* refactor(ui): name the stdio banner conditions on the server detail page
Keeps local/no-long-condition-chain within its budget
* fix(proxy): log the ignored DB-stored LITELLM_ENABLE_MCP_STDIO warning once
The DB config sync re-reads environment_variables on every cycle, so a stored
flag logged the warning on each sync per worker
(cherry picked from commit
|
||
|
yassin
|
84e85778ba |
fix(proxy): enforce the migration check by default and stop building SpendLogs indexes in migrations
ENFORCE_PRISMA_MIGRATION_CHECK now defaults to true in both the proxy CLI and the standalone migration entrypoint, with ENFORCE_PRISMA_MIGRATION_CHECK=false or --no-enforce_prisma_migration_check as the opt-out. The two LiteLLM_SpendLogs index migrations become SELECT 1 so operators build those indexes online by hand. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
5663d07a1a
|
revert: backport of #43948, #44109, #44124 to rc/1.104.0 (#44156) (#44212)
This reverts merge commit |
||
|
yassin
|
f46ddd24aa |
feat(proxy-extras): make the SpendLogs index build opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
7f81c8ea88 |
fix(proxy-extras): bound the lock waits of the partitioned SpendLogs index build (#44109)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
24e0d53fc2 |
fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948)
The two SpendLogs index migrations shipped in v1.103.0 each break one table shape: the plain CREATE INDEX holds a SHARE lock on a large unpartitioned table and the CONCURRENTLY one fails with 0A000 on a partitioned parent. Both files are now inert and the indexes are built by a table-driven, shape-aware step after migrate deploy: CONCURRENTLY on a plain table, ON ONLY the parent plus per-partition CONCURRENTLY and ATTACH PARTITION on a partitioned one. The migration job builds them synchronously and exits non-zero on failure; a serving proxy that ran migrate deploy itself builds them in the background off the readiness path. A valid index of the same definition under another name is renamed and reused, an invalid one is rebuilt, and extra copies are reported with their DROP INDEX statement instead of being dropped. The migration checker rejects any CREATE INDEX on LiteLLM_SpendLogs or LiteLLM_ErrorLogs in future migrations
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit
|
||
|
yuneng-jiang
|
61876f228e
|
fix(proxy): restore pre-config-wins handling of pass-through endpoints (#43962) (#44054)
* fix(proxy): restore pre-config-wins handling of pass-through endpoints
Config-wins (#41779) made general_settings.pass_through_endpoints a config-owned key. The DB reader then got the config list back as if it were DB rows, re-registered each entry without forward_headers on every DB sync, and the stripped copy won the route lookup, so a config pass-through with forward_headers: true stopped forwarding Authorization. UI create, update and delete of pass-throughs were also rejected while the config declared any.
This puts pass-throughs back on their pre-#41779 path: the settings store no longer lets the config own the key, the config list is captured env-resolved at load_config, each DB sync merges DB entries with config entries on paths the DB does not declare, and /config/field/info reads the stored rows only. A UI pass-through write re-applies that merge immediately so the config entries stay served until the next sync.
* fix(proxy): keep config pass-throughs in every reload of the merged list
get_config now returns DB pass-throughs plus config ones on other paths,
each DB sync republishes that merged list, and /config/field/info reads
pass_through_endpoints from the DB row so a UI write never drops stored
entries when models are not stored in the DB
* fix(proxy): keep serving pass-throughs while the config file reloads
load_yaml cleared the runtime pass-through list, so auth: false routes
answered 401 while get_config awaited the database
* fix(proxy): read stored pass-throughs from the writer before a UI write
A lagging read replica could return an older list, and the UI create and
edit flows write the whole field back
* fix(proxy): apply config file pass-through auth changes on reload
The kept runtime list was merged as if it were DB entries, so an edited
config entry on the same path was dropped. Merge the stored DB row with
the fresh config instead, and give the field-info test mock a writer
* fix(proxy): keep pass-throughs served while a DB sync reads the database
get_config resets the stored DB rows before reading them again, which
cleared the served pass-through list and made auth: false routes answer
401 for the length of the read
* refactor(proxy): move the settings store reload out of the loop
basedpyright rejects a Final variable assigned inside a loop
(cherry picked from commit
|
||
|
yuneng-jiang
|
29c35ba3c3
|
chore(release): sync rc/1.104.0 to v1.104.0-rc.2 (#43825)
* refactor(auth): bind UI/CLI session tokens to their own AES-GCM context Backport of BerriAI/litellm-private#5 (12981f93d3) onto rc/1.104.0, applied as the PR's net diff so main-only intermediate refactors stay out. * chore(lint): scope a TRY004 suppression to the bearer-token salt key check This line's ruff strict budget has no headroom for the one TRY004 the backport adds; the raise reports missing configuration, not a bad type. |
||
|
yuneng-jiang
|
a76395ec70
|
fix(proxy): unregister logging callbacks removed from the stored config (#43429) (#43432)
* fix(proxy): unregister logging callbacks removed from the stored config
POST /config/callback/delete saved the config and resynced, but the resync only
ever added callbacks, so a deleted callback kept exporting and kept showing in
/get/config/callbacks as read-only on every worker.
ProxyConfig now tracks which callback list entries each DB config sync
registered and unregisters them once the stored config stops listing them.
Callbacks it did not register (YAML, code) are never touched, and a failed
config load skips the sync instead of treating the config as empty.
* refactor(proxy): keep callback sync comprehensions to one for clause
(cherry picked from commit
|
||
|
yuneng-jiang
|
1df3f10641
|
test(unit): stop test modules from putting their own directory on sys.path (#43420)
test_ssl_verify_unit.py inserted tests/unit at the front of sys.path, so any later import of litellm_proxy_extras resolved to the tests/unit/litellm_proxy_extras test package. Whenever the CircleCI shard split collected that file before test_litellm_proxy_extras_logging.py, collection failed with ModuleNotFoundError. test_gemini_session_leak.py had the same insert for its own directory |
||
|
yuneng-jiang
|
d0e417e3c8
|
test(e2e): skip the LangSmith batch serialization e2e on rc/1.104.0 until CI has a LangSmith key (#43418)
* test(e2e): skip the LangSmith batch serialization e2e until CI has a LangSmith key LANGSMITH_API_KEY is not provisioned in the e2e CI environment, so the test hard-fails on every run * test(e2e): only skip the LangSmith e2e when LANGSMITH_API_KEY is missing |
||
|
yuneng-jiang
|
2b6db94281
|
test(e2e): report batch cleanup leftovers as a plain UserWarning (#43406)
The leftover warning used a class defined in a test-directory module. The xdist controller cannot import it, so an uncaught leftover warning crashed the whole e2e run. Same change as #43391 on rc/1.103.0 |
||
|
yuneng-jiang
|
e0cbb0db82
|
fix(streaming): keep litellm Usage on text-completion usage chunks (#43047) (#43402)
* fix(streaming): keep litellm Usage on text-completion usage chunks
* fix(streaming): convert provider usage to litellm Usage instead of dropping it
(cherry picked from commit
|
||
|
yuneng-jiang
|
6d672483c9
|
revert(usage): remove the top-N key cap, its follow-ups and the daily global spend rollup code from rc/1.104.0 (#43385)
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
* Revert "feat(proxy): server-side Team Usage export beyond the top-N key cap (#42996)" This reverts commit |
||
|
yuneng-jiang
|
95f33f477d
|
chore: backport CircleCI speedups (#43347) and cost-map fixes (#42951) to rc/1.104.0 (#43366)
* fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) (cherry picked from commit |
||
|
yuneng-jiang
|
822ccfc11d
|
fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) (#43354)
* fix(s3_v2): drop terminal upload failures, bound retries per flush and enforce the queue cap at enqueue
* fix(s3_v2): keep retrying credential-rotation 403s, only AccessDenied-style errors are terminal
* test(env_keys): exclude DEFAULT_S3_MAX_FLUSH_ATTEMPTS as an internal tuning var
* fix(s3_v2): retry every 5xx, warn on first queue overflow, validate the flush budget
* fix(s3_v2): read the queue cap defensively so un-initialized loggers still enqueue
* fix(s3_v2): drop the getattr in _enqueue and tighten the retry tests
* fix(s3_v2): keep the constructor flush budget when the callback override is invalid
* feat(s3_v2): adapt per-object upload concurrency to sink latency and throttling
* refactor(s3_v2): tidy adaptive limiter
* perf(s3_v2): wake one waiter per released upload slot
* feat(s3_v2): make the enqueue queue cap configurable with s3_max_queue_size
* test(s3_v2): audit cells for cache hits, coded 403 and callback modes
* fix(s3_v2): retry bucket-wide failures by default, age-budget requeues and make terminal drops and adaptive concurrency opt-in
* refactor(s3_v2): suppress the missing-waiter ValueError explicitly in the adaptive limiter
* fix(s3_v2): count oldest events trimmed after a failed flush as callback failures
* style(s3_v2): move the mutable-ok marker onto the list literal it suppresses
* fix(s3_v2): report post-flush overflow drops once and grow adaptive concurrency above the floor before asserting back-off
* test(s3_v2): fail the SlowDown back-off test when the measured window sees no PUTs
* fix(s3_v2): restore base retry defaults, opt-in age budget, no enqueue cap, back off outside the limiter slot
* fix(s3_v2): hoist the default no-op upload slot to a module constant
* fix(s3_v2): drop unused mutable-ok suppressions on queue appends
* fix(s3_v2): keep the retry queue oldest-first and prioritise fresh events at upload time
* style(s3_v2): keep the mutable-ok marker on the queue list literal
* fix(s3_v2): build request bodies inside the upload slot and keep the sync retry set at base parity
* test(s3_v2): drop wall-clock sleeps from the unit tests
* fix(s3_v2): rebuild the request body inside the slot on every retry attempt
* test(s3_v2): anchor the backoff window on the first observed failure and tighten shard assertions
* fix(s3_v2): default the upload slot to the logger limiter so monkeypatched doubles keep working
* test(s3_v2): clear ambient AWS env credentials so the rotating profile signs the sync retry test
* test(s3_v2): shrink the linear send-batch perf test to 2k/8k elements
* test(s3_v2): fix stale batch sizes in the perf test assert message
* fix(s3_v2): keep async in-call retries on the base 403/500/503 set
* fix(s3_v2): hold the upload slot across retries, restore the bool upload contract, and fail safe on bool config
* fix(s3_v2): mark dropped uploads by element identity so a shared key cannot mask a retryable sibling
* fix(s3_v2): make the per-flush drop lookup constant time
* fix(s3_v2): take the upload slot in the caller like base, build the body once per attempt loop
* fix(s3_v2): match base retry, logging and hook behaviour unless the new options are opted in
* test(s3_v2): assert the wire key in the init-bypassed sync upload test
* test(s3_v2): assert the signed headers and wire key in the init-bypassed sync upload test
* fix(s3_v2): default the upload limiter at class level instead of reading it with getattr
* fix(s3_v2): drop the duplicate annotations that redeclare the class-level counters
* feat(s3_v2): drop terminal-failed uploads by default and bound retry age to one hour
* fix(s3_v2): fall back to the configured retry age on invalid values
* test(s3_v2): drive retry-age tests from a fixed clock
* test(s3_v2): audit cells for retry-age opt-out and 429 single-put parity
---------
(cherry picked from commit
|
||
|
yuneng-jiang
|
7248fd8328
|
chore: backport CI fixes, #43206 and Rust changes to rc/1.104.0 (#43338)
* feat(model-catalog): add Rust registry validation (#43136) * doc * feat(model-catalog): add Rust registry validation * test(model-catalog): ignore integration tests * docs(model-catalog): fix validation note grammar --------- Co-authored-by: Yujong Lee <yujong@berri.ai> (cherry picked from commit |
||
|
yuneng-jiang
|
4b3c72ac0e
|
test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) (#43267)
test_update_config_success_callback_normalization replaced
proxy_server.proxy_logging_obj with a MagicMock and never restored it.
Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping,
end-user and MCP tests on the same xdist worker awaited that mock and
failed. The test now uses monkeypatch.
test_prometheus_logging_callbacks set verbose_logger to DEBUG and
litellm.set_verbose at import, so every worker in the unit job ran with
DEBUG on. That broke caplog equality in the JEV classifier test, the
vertex streaming memory ratio, and four event-loop lag checks. The
module-level setup is removed; nothing in the file depended on it.
#43081 removed the OCR harness modules but left them in the
importability parametrize list.
test_get_model_info_bedrock_region reassigned litellm.model_cost and set
LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared
the get_model_info caches, so it failed whenever an earlier test had
looked up the regional model. It now uses monkeypatch and invalidates
the caches; the local_testing isolation fixture also invalidates them
after restoring model_cost.
The Windows job hit CircleCI's 10 minute no-output limit while cargo
compiles the Rust crates inside uv sync and uv build. Those two steps now
allow 30 minutes of silence.
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
c19ce71bcd
|
fix(presidio): mask streamed /v1/messages output when the first upstream read is a keepalive, a data-less ping, or a split utf8 character (#43023)
* test(presidio): cover first-frame utf8 split, comment keepalive and data-less ping in streaming output masking Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(presidio): classify the streaming output shape on a frame with a data line and tolerate a split utf8 boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(presidio): relay leading data-less sse frames before classifying the stream shape Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
2701e2008b
|
fix(ui): group cost optimization cache leakage by model group (#43008)
* test(ui): cover cost optimization cache leakage grouping by model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): group cost optimization cache leakage by model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8b3939da25
|
fix(key_management): count team unified access group MCP servers when validating key MCP grants (#41231)
* fix(key_management): count team unified access group MCP servers when validating key MCP grants Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop explanatory suffix from pyright suppression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(key_management): restore reason on pyright suppression for LIT004 gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(key_management): assert union result and suppress TQ008 on litellm-internal patches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: verify unified MCP team grants through real resolvers --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
e319bf270c
|
feat(langfuse): migrate the sdk callback to langfuse v4 (#36741)
* feat(langfuse): migrate the sdk callback to langfuse v4
Replace the v2 trace()/generation()/span() calls with SDK v4 observations exported over OpenTelemetry, with one isolated tracer provider per Langfuse credential set, a discarding exporter for mock mode, and v4 trace and observation id normalization. Keeps the session-header trace provenance logic from main so each call under a session alias still gets its own trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): drop the always-true prompt client check now that v4 get_prompt is non-optional
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): isolate the e2e sync test from cached clients and log the real sdk major
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the slack trace-url lookup and drop dead v2 test shims
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(slack): cover the langfuse trace url built from the logger host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): pin langfuse to the locked 4.15.2 in the pip image
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): hash all-zero trace and observation ids instead of passing them through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(langfuse): honour caller generation ids and assert v4 OTLP exports in legacy tests
v2 accepted generation(id=...). v4 derives the observation id from the OTel
span id, so the isolated tracer provider now carries an id generator that
hands out the id start_generation asked for through a context variable, and
the callback passes the resolved generation_id metadata into it.
The legacy e2e suite patched httpx.Client.post and compared v2 ingestion
batches; it now patches requests.Session.post, decodes the OTLP protobuf
and compares the exported generation against regenerated fixtures. The
local readback test replaces the removed get_generations() with
api.observations.get_many() and polls Langfuse Cloud instead of sleeping.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): read the sdk version header from package metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): propagate trace_metadata as trace-level attributes in v4
v2 wrote trace(metadata=...) onto the trace object. In v4 the trace only
carries what the observations propagate, so a continuation request with
update_trace_keys=["trace_metadata"] updated the generation's metadata
while the trace kept its stale values. Coerce each entry to the SDK's
string limit and hand it to propagate_attributes(metadata=...).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): propagate interrupts raised during deferred client teardown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): honor ssl_verify=False and SSL_VERIFY on the v4 OTLP exporter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): fall back to the default CA when the configured bundle path is missing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): renew the client when eviction lands before the callback lease
The cache can evict a logger between handing it to the callback and the callback taking its
lease. Such a lease now hands back a fresh client acquired through the same parameters, so that
callback exports through a live tracer provider instead of one teardown already shut down.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): emit litellm_call_id and response_id as generation metadata
v2 put the provider response id inside the generation id. v4 observation ids are 16 hex chars derived from that string, so the ids move to generation metadata to keep generations searchable by response id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): read the response id through a typed protocol
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): do not claim trace root when continuing an existing trace
Langfuse derives a trace's name and I/O from any observation flagged
langfuse.internal.as_root, so a request carrying existing_trace_id
renamed the trace to the generation name and replaced the trace input
and output on every continuation. v2 only updated the keys listed in
update_trace_keys. Continuations now export as plain children of the
remote parent and keep the explicit langfuse.trace.* attributes for the
fields they do want changed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): iterate lease renewal instead of recursing, monkeypatch update_trace_keys flag in tests
The recursive lease fallback tripped tests/code_coverage_tests/recursive_detector.py; the renewal
candidates are now walked with itertools.chain. The six update_trace_keys tests set the litellm
global through pytest monkeypatch so the TQ008 budget stays within its ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): retry raised OTLP exports and honor LANGFUSE_TIMEOUT
The OTLP http exporter only retries 429 and 5xx; a connect or read timeout
propagates and BatchSpanProcessor drops the batch. Wrap the exporter in
RetryingSpanExporter (three backoff retries, as the v2 consumer did) and
build it on every path so the default and private-CA deployments share the
same channel, timeout and retry behaviour
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): sample on a hash of the full trace id and tolerate bad LANGFUSE_SAMPLE_RATE
TraceIdRatioBased reads the low 64 bits of the trace id. litellm trace ids are
UUIDs, whose variant bits sit at the top of that word, so every fractional rate
up to 0.5 dropped all traces. A SHA-256 of the full id gives an unbiased,
deterministic decision. Values outside [0, 1] or non numeric now warn and export
everything instead of raising during callback construction, which surfaced as a
500 on the first request of each worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): put the Langfuse trace link back into Slack alerts
The proxy registers LangfusePromptManagement for callbacks: ["langfuse"], so the alert helper never saw the literal "langfuse" string and returned before looking up the trace id, and the prompt management logger never stored the trace id it got back from log_event_on_langfuse. Recognize LangFuseLogger instances in the callback list, record the returned trace id in the shared service trace id cache, and skip the link when no trace id arrives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): relock langfuse 4.15.2 and opentelemetry 1.33.1 on current main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(langfuse): mark the deliberate blind except in client teardown for the strict ruff gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): pass the resource attributes mapping straight to Resource.create
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): warn about ignored UPSTREAM_LANGFUSE_* on the shared client init path too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): normalise the OTLP export path so a trailing host slash never yields a double slash
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): nest guardrail and grounding spans under the generation
Langfuse v4 derives the trace name and I/O from every observation marked as_root, and the one with the latest start time wins. Guardrail and grounding spans used to claim root next to the generation, so a post_call guardrail could replace the model's request and response on the trace with its own. Only the generation claims root now; the sibling spans become its children
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): rebuild the cached bundle when mock mode or sample rate changes
The SDK keys resource bundles on the public key alone, so a bundle built with the discarding exporter for LANGFUSE_MOCK, or with an earlier LANGFUSE_SAMPLE_RATE, was handed back to a client that asked for a live exporter or a different rate. Compare both when deciding whether the cached bundle is still valid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): keep trace_public true when a guardrail span is exported
Langfuse folds langfuse.trace.public across every observation in the trace and reads a missing attribute as false, so a guardrail child span without the flag turned a trace_public: true request private on Langfuse Cloud. Child spans now repeat the generation's value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): emit observations as plain OTel spans, keep the SDK for prompts and auth
The callback now owns an isolated TracerProvider and OTLP exporter and builds generation and child spans with public OpenTelemetry APIs plus the LangfuseOtelSpanAttributes constants. Caller trace ids, generation ids, parent observation ids and historical start and end times are honoured through the OTel id generator, remote SpanContext and explicit span timestamps, so no private Langfuse SDK tracing handle is used any more. The Langfuse client stays only for get_prompt and auth_check
This also resolves the gauntlet findings on the previous draft: fresh traces start from an empty context so caller application spans are never stamped, the Slack trace link is read from the request logging state instead of constructing a logger per alert, a truthy non-mapping trace_metadata is serialized instead of raising, trace_input and trace_output land on the root generation, discarding a cached client is done under the lock, and the prompt cache no longer leaks a task manager because the client cache no longer tears down shared providers
Fixtures under tests/logging_callback_tests lose the SDK-private langfuse.internal.as_root marker; every other exported attribute is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): hand the SDK client a validated sample rate so an unusable LANGFUSE_SAMPLE_RATE no longer breaks the callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): gate the SDK version before importing the OTel module in prompt management
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush every export channel on proxy shutdown and use the callback's host in Slack trace links
The shutdown hook imported litellm.utils.langFuseLogger, a global the callback registry never assigns, so a graceful restart dropped the spans still queued in the batch processors. Shutdown now calls flush_langfuse_tracing, which force-flushes every acquired channel. The Slack alert link falls back to the registered LangFuseLogger's langfuse_host when the request carries no dynamic host, and the export endpoint tests pin that scheme-relative or absolute LANGFUSE_OTEL_TRACES_EXPORT_PATH values stay on the configured host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): store resolved credentials on LangfusePromptManagement
The Slack alert trace link reads langfuse_host from every registered LangFuseLogger. Prompt management subclasses it without calling the parent constructor, so it never set the attribute and the alerting handler crashed before posting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush every export channel concurrently under one shutdown deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush export channels on daemon threads so a stuck channel cannot hold up interpreter exit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): own the tracer config and drop the SDK client for prompts and auth
The callback's TracerProvider now sets its sampler, span limits and id generator explicitly so unrelated OTEL_* variables no longer change what Langfuse receives, and trace metadata is written once on the trace instead of folded into the generation, which kept input and output under the attribute cap. Spans are emitted under the langfuse-sdk scope so Langfuse renders them natively, the batch processor queues 100k spans and honors LANGFUSE_FLUSH_AT, and the proxy shutdown flush runs off the event loop with a 10s deadline and logs a miss.
Prompts, auth_check and the project id now go through LangfuseAPI directly with a litellm-owned TTL cache, so no Langfuse() client is built and a host application's client on the same public key is left alone. Dead attributes, the unreachable exporter branch and the export list are cleaned up, and the client-budget eviction behavior is documented.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): export OTLP spans and fetch prompts through litellm's HTTPHandler instead of a private requests session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): gate the SDK version before importing the tracing module and retire unheld export channels
An installed v2 SDK used to fail inside the langfuse_sdk import and surface as "Langfuse not installed"; the version check now runs first so v2 users get the upgrade message, and only PackageNotFoundError means the package is missing
Export channels are now leased per credential set: acquire adds a holder, LangFuseLogger.stop (called by DynamicLoggingCache on expiry) releases one, and a channel with no holders is flushed and shut down after a 60 s grace, so rotating key or team credentials no longer grows one batch thread per credential set for the life of the process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): end the generation when a child span fails, take the client slot last, keep prompt cache keys structured
Generation spans now end in a finally block so a bad guardrail or provider entry cannot strand the trace. The logger acquires its export channel and REST client before counting a client slot and releases the channel synchronously if the REST client fails to build, so retries after a bad config do not exhaust the budget. LANGFUSE_TIMEOUT accepts decimals for the REST client like it already did for OTLP export. The prompt cache keys on (name, version, label) so a missing label and the literal label None stay apart
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): claim the cache entry before releasing its slot and channel hold on eviction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): coerce generation names, keep v2 release, timeout and retry defaults, refresh stale prompts off the loop
A non-string metadata generation_name reached the OTLP encoder and took the whole batch down; it is now exported as its text and the exporter drops only the span the encoder rejects. LANGFUSE_RELEASE falls back to the deploy platform's commit variable again, the export deadline is back to the v2 default of 20 s and LANGFUSE_MAX_RETRIES sizes the retry ladder. An expired prompt is served at once while one background thread refreshes it, a re-acquired export channel cancels the pending retire timer, and flush reports delivery rather than a drained queue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the current Langfuse shutdown flush warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): keep host OTel resource out, carry big metadata ints, tolerate bad flush and TTL env, stamp trace I/O under a parent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): name a malformed prompt cache TTL before the SDK import, keep metadata ints JSON safe, retry every 5xx export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): name the auth check failure, split a 413 export, wire LANGFUSE_DEBUG, stamp error output under a parent, send the ingestion version header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): honor LANGFUSE_DEBUG on the callbacks path, cap retry backoff, name the auth failure status and body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): cap LANGFUSE_MAX_RETRIES at 1000 so an absurd value cannot stall callback init
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): fold 413 halving into bounded rounds instead of recursion
The code-quality recursive-function gate flagged LangfuseSpanExporter.export. A batch of n spans settles within n.bit_length() halving rounds, so the split is a reduce over a frozen round state with the same posts, logs and results. The TTL gate test now asserts the gate returns without raising
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): truncate a single oversized span like v2 instead of dropping it, no retries on REST auth and project lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): write the metadata truncation marker under a flattened key so Langfuse keeps it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): patch the HTTPHandler export path and sync the metadata fixture and lease registry with the v4 callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): give the 413 split helpers a single explicit return path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): url-encode prompt names and fetch cold prompts without client retries
A cold get_prompt runs inline on the event loop; the generated v4 client's default two retries slept through
Retry-After (up to 60 s per attempt) and held the loop. The wrapper also passed the raw name into
api/public/v2/prompts/{name}, so 'what?' fetched prompt 'what' and folder names left the route. Quote the
name with safe='' like the v4 SDK's own get_prompt and pass max_retries=0 like the projects.get calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): retry a cold prompt miss once and drop upstream headers from prompt errors
A cold prompt fetch makes one immediate second attempt after a 5xx or a
transport failure, as the v2 client did, still with the generated client's
sleeping retries and Retry-After handling off so the event loop never stalls.
A failed fetch raises LangfusePromptError carrying only the status and body,
so the proxy no longer forwards Langfuse's response headers to its client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): stub the logger in the health auth_check test instead of dialing a closed port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): integration test for OTLP v4 delivery and prompt fetch through a real proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): keep the pip image's langfuse and otel pins on the v2 line its litellm 1.83.0 wheel expects
The image validates the published PyPI artifact, whose langfuse callback still
reads langfuse.version, so the 4.15.2 pin broke that callback. The pins move
together with the next LITELLM_VERSION bump. Also rewords the trace_version
precedence test docstring: v2 carried two version fields, v4 has one per span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
ccf866c801
|
fix(router): await budget redis pipeline before sync reads (internal copy of #32618) (#43125)
* fix(router): await budget redis pipeline before sync reads * refactor(router): remove superseded Redis flush helper * fix(router): preserve concurrent spend during Redis synchronization * fix(router): type per-key spend totals without loop Final bindings * fix(router): log Redis failures before cancellable cleanup * fix(router): finalize Redis batches before propagating cancellation * test(router): reproduce cancellation while Redis cleanup is blocked * test: align budget hotpath checks with awaited Redis flush * fix(budgets): finish Redis flush after cancellation while queued * test(budgets): consolidate Redis regressions in mapped tests * fix(budgets): coalesce failed Redis increments by key * test(budgets): assert spend behavior instead of batch state * perf(router): sum pending budget spend by key once per sync --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |