litellm/litellm
Zachary Lyon 19c13ac971
feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099)
* feat(proxy): add TinyFish Agent API passthrough with per-step billing

* chore(ui): regenerate dashboard API types for /tinyfish passthrough

* fix(proxy): satisfy strict lint budget for tinyfish passthrough

* style: ruff format tinyfish passthrough handler

* test(proxy): exercise tinyfish route through the app with a faked upstream

* refactor(proxy): make cross-module tinyfish billing hooks public

* fix(proxy): tolerate transient tinyfish poll failures instead of dropping the charge

* fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller

* Revert "fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller"

This reverts commit ef0bcfb4a0.

* fix(proxy): bill tinyfish SSE runs via detached poller and only COMPLETED runs

Disconnected run-sse clients previously left completed runs unbilled: the stream-end
handler saw a still-RUNNING run and logged $0. The poller now spawns from the streaming
path on the first run_id frame, outlives the disconnect, and writes the one spend row when
the run turns terminal; the stream-end path only logs the $0 fallback for run_id-less
streams. Costs now apply only to COMPLETED runs ($0 for FAILED/CANCELLED, matching the
upstream invoice), spend rows carry the request's litellm_call_id (previously NULL
request_ids collided and were silently dropped), and the GET /v1/runs listing is blocked
so callers behind the shared key cannot discover each other's runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* fix(proxy): drop GET /v1/runs from the tinyfish allowlist error message

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* chore(proxy): sync openapi artifacts for tinyfish docstring, suppress LIT011 on flag write

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* style(proxy): ruff-format the sse poller flag write

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* style(proxy): keep the rebind-ok suppression on the flag write's own line

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* fix(proxy): harden tinyfish billing paths from review findings

Skip failure dispatch when the SSE poller owns billing (a failure row collided with the
poller's billed row on request_id and dropped the charge), late-spawn the poller for
run_ids that arrive in unterminated frames instead of mispricing RUNNING runs at $0,
thread litellm_params into poller-billed standard logging objects so SLO consumers see
attribution, untype the run error field so upstream error-shape drift cannot void a
billable run, normalize a schemeless TINYFISH_AGENT_API_BASE, extend the poll budget to
cover queue wait (3600s) with ~60s outage tolerance, and log poller cancellation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* chore(proxy): satisfy ratcheted BLE001/LIT002 budgets from main in tinyfish handler

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* style(proxy): drop stray blank line from merge resolution

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* fix(proxy): reject passthrough envelope controls on tinyfish route, raise blocking-run timeout

The generic passthrough unwraps a caller-supplied custom_body as the forwarded request and
honors a caller stream flag, so custom_body.use_vault bypassed the credentialed-run 403 and
stream: true flipped a blocking run into the streaming pipeline. The route now 400s the
envelope fields (custom_body, stream, query_params); streaming comes from the endpoint.
Blocking runs also get a 1500s default timeout covering the upstream 1200s run cap, unless
the operator configured pass_through_request_timeout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* style(proxy): resolve operator timeout without a dict-literal default

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4

* test(passthrough): list the TinyFish route among protocol-constrained pass-through routes

* chore(proxy): regenerate the lazy OpenAPI snapshot after merging main

* chore(proxy): keep the lazy OpenAPI snapshot as CI's Python 3.12 renders it

* fix(proxy): reject TinyFish POST bodies that are not a JSON object

A form-encoded or text body carried stream and use_vault past both field
gates, because the gates only saw fields the body parsed to as JSON. The
route now checks the content type before reading the body and answers 400
for anything that is not a JSON object.

* fix(tinyfish): reject submit paths with extra slashes so run-async always bills

The allowlist dropped empty path segments, so POST /v1/automation/run-async/
was forwarded upstream while the billing dispatch only recognises the exact
path and would have logged the submit at $0 without starting the poller.
Any path with a trailing or doubled slash now returns 403 before forwarding.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 21:21:43 -07:00
..
a2a_protocol Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
anthropic_interface feat(proxy): opt-in litellm_call_id in JSON error bodies (#42391) 2026-09-21 19:17:18 -07:00
assistants
batch_completion
batches Merge remote-tracking branch 'origin/main' into litellm_mistral_ocr_batches 2026-09-18 21:51:33 -07:00
caching Merge origin/main into litellm_rust_qdrant_semantic_cache 2026-09-21 22:56:07 +00:00
chat_completions feat(rust-bridge): add cache and secret migration foundations (#42328) 2026-09-22 03:41:04 +00:00
completion_extras Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
compression merge: main into litellm_headroom_protect_cached_prefix 2026-09-15 04:12:28 +00:00
containers docs: stop advertising sk-1234 as the master key in shipped configs and examples 2026-09-19 12:59:48 -07:00
endpoints/speech/speech_to_completion_bridge
evals
experimental_mcp_client fix(mcp): paginate optional discovery lists 2026-09-21 11:37:55 -07:00
files Merge remote-tracking branch 'origin/main' into litellm_mistral_ocr_batches 2026-09-18 21:51:33 -07:00
fine_tuning
google_genai fix(google_genai): drop non-object tool parameters instead of forwarding them 2026-09-19 19:19:33 -07:00
images Merge branch 'main' into litellm_add_edenai_provider 2026-09-21 19:34:49 +00:00
integrations feat(cost): warn and count $0 cost on billable requests (#42345) 2026-09-21 19:49:26 -07:00
interactions refactor(types): replace Any with proven types in 34 files 2026-09-20 10:16:48 +00:00
litellm_core_utils fix(responses): drop client_metadata and merge system messages for Databricks chat-only models (#42390) 2026-09-21 20:09:54 -07:00
llms fix(anthropic_adapter): keep reasoning_effort a string for targets that stay on chat completions (#42401) 2026-09-21 21:08:28 -07:00
messages feat(rust-bridge): add cache and secret migration foundations (#42328) 2026-09-22 03:41:04 +00:00
models feat(auth): breached password detection, self-service change-password and forced password reset 2026-09-21 18:48:35 +00:00
ocr feat(rust-bridge): add cache and secret migration foundations (#42328) 2026-09-22 03:41:04 +00:00
passthrough Merge pull request #41557 from BerriAI/litellm_azure_speech_passthrough 2026-09-18 14:05:55 -07:00
proxy feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099) 2026-09-21 21:21:43 -07:00
proxy_auth
rag fix(types): read upstream headers through a typed helper 2026-09-21 13:16:58 -07:00
realtime_api Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
repositories Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
rerank_api
responses feat(rust-bridge): add cache and secret migration foundations (#42328) 2026-09-22 03:41:04 +00:00
router_strategy feat(openrouter): price typesafe/jev-1.13 and add an openrouter decisions pass-through (#42301) 2026-09-22 02:44:15 +00:00
router_utils Merge pull request #42283 from BerriAI/litellm_mid_stream_fallback_walks_full_list 2026-09-21 15:14:04 -07:00
rust_bridge feat(rust-bridge): add cache and secret migration foundations (#42328) 2026-09-22 03:41:04 +00:00
sandbox
search
secret_managers Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
skills
types feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099) 2026-09-21 21:21:43 -07:00
vector_store_files
vector_stores refactor: replace Any with precise types across 54 modules 2026-09-14 11:13:39 +00:00
videos
__init__.py feat(fal_ai): add flux-lora-depth image edits and moondream3 chat completions (#42334) 2026-09-21 19:01:28 -07:00
_internal_context.py
_lazy_imports.py
_lazy_imports_registry.py feat(fal_ai): add flux-lora-depth image edits and moondream3 chat completions (#42334) 2026-09-21 19:01:28 -07:00
_logging.py refactor(types): replace Any with proven types in 32 files 2026-09-21 10:51:46 +00:00
_redis.py fix(redis): accept every truthy flag and sign serverless ElastiCache caches 2026-09-10 10:13:03 -04:00
_redis_credential_provider.py fix(redis): accept every truthy flag and sign serverless ElastiCache caches 2026-09-10 10:13:03 -04:00
_service_logger.py
_uuid.py
_version.py
anthropic_beta_headers_config.json fix(anthropic): map the dangerous-tool-use beta for Bedrock Mantle so safeguards never reach it without the beta 2026-09-21 13:24:52 -07:00
anthropic_beta_headers_manager.py fix: drop a blank anthropic-beta header before it reaches the provider 2026-09-19 20:13:59 -07:00
blog_posts.json
budget_manager.py
constants.py Merge pull request #41213 from BerriAI/litellm_spend_log_cleanup_cancel_outcome 2026-09-21 17:03:25 -07:00
cost.json
cost_calculator.py feat(cost): warn and count $0 cost on billable requests (#42345) 2026-09-21 19:49:26 -07:00
exceptions.py feat(proxy): add budget_exceeded_status_code setting to restore 429 for budget refusals 2026-09-20 07:00:10 +00:00
main.py feat(fal_ai): add flux-lora-depth image edits and moondream3 chat completions (#42334) 2026-09-21 19:01:28 -07:00
model_prices_and_context_window_backup.json chore(prices): sync OpenRouter prices: 2 models, 2 deprecated (#42381) 2026-09-21 21:11:22 -07:00
policy_templates_backup.json
provider_endpoints_support_backup.json Merge pull request #41101 from hMED22/litellm_add_edenai_provider 2026-09-21 16:16:28 -05:00
py.typed
router.py fix(router): wrap every Responses and Messages fallback hop for mid-stream failover 2026-09-21 14:19:52 -07:00
scheduler.py
setup_wizard.py
timeout.py
utils.py fix(proxy): release unclaimed budget reservations at request end (#42304) 2026-09-21 19:51:12 -07:00