scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-06 02:48:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

Merge pull request #25616 from BerriAI/main

Browse source 
merge main
This commit is contained in:
Sameer Kankute 2026-04-13 08:43:43 +05:30 • committed by GitHub
commit fa605d85c0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
583 changed files with 19121 additions and 4730 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.github/pull_request_template.md vendored
View file

@ -32,6 +32,13 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
- [ ] **Merge / cherry-pick CI run**
Links:
## Screenshots / Proof of Fix
<!-- Include screenshots, screen recordings, or log output demonstrating that your changes work as expected.
For bug fixes: show reproduction before the fix and passing behavior after.
For new features: show the feature working end-to-end.
For UI changes: include before/after screenshots. -->
## Type
<!-- Select the type of Pull Request -->

2
README.md
View file

@ -12,7 +12,7 @@
</a>
</p>
</p>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://litellm.ai/enterprise"target="_blank">Enterprise Tier</a> | <a href="https://litellm.ai/" target="_blank">Website</a></h4>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://litellm.ai/enterprise"target="_blank">Enterprise Tier</a> | <a href="https://www.litellm.ai/ai-gateway" target="_blank">Website</a></h4>
<h4 align="center">
<a href="https://pypi.org/project/litellm/" target="_blank">
<img src="https://img.shields.io/pypi/v/litellm.svg" alt="PyPI Version">

8
docker/Dockerfile.custom_ui
View file

@ -71,8 +71,16 @@ WORKDIR /app
RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh
RUN sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
# Run as non-root user
RUN groupadd --gid 1000 appuser && useradd --uid 1000 --gid 1000 --no-create-home appuser \
&& chown -R appuser:appuser /app
USER appuser
# Expose the necessary port
EXPOSE 4000/tcp
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health')"]
# Override the CMD instruction with your desired command and arguments
CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug"]

8
docker/Dockerfile.health_check
View file

@ -13,12 +13,12 @@ RUN pip install --no-cache-dir -r requirements.txt
RUN chmod +x /app/health_check_client.py
# Run as non-root user
RUN adduser --disabled-password --gecos "" --uid 1001 healthcheck
USER healthcheck
RUN groupadd --gid 1000 appuser && useradd --uid 1000 --gid 1000 --no-create-home appuser
USER appuser
# Health check
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD python /app/health_check_client.py --help || exit 1
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
CMD ["python", "/app/health_check_client.py", "--help"]
# Set entrypoint
ENTRYPOINT ["python", "/app/health_check_client.py"]

1
docs/my-website/blog/april_townhall_announcement/index.md
View file

@ -4,6 +4,7 @@ title: "April Townhall: Security + Product Roadmap"
date: 2026-04-02T07:30:00
authors:
- krrish
- ishaan-alt
description: "Join the LiteLLM April townhall on Friday, 10 April at 7:30 AM to learn about LiteLLM's security and product roadmap."
tags: [announcement, townhall]
hide_table_of_contents: true

162
docs/my-website/blog/april_townhall_updates/index.md Normal file
View file

@ -0,0 +1,162 @@
---
slug: april-townhall-updates
title: "April Townhall Updates: CI/CD v2, Stability, and Product Roadmap"
date: 2026-04-10T12:00:00
authors:
- krrish
- ishaan-alt
description: "A recap of the April LiteLLM town hall covering CI/CD v2, product stability work, and the near-term roadmap."
tags: [townhall, security, reliability, product]
hide_table_of_contents: false
---
import Image from '@theme/IdealImage';
Thank you to everyone who joined our April town hall.
We used the session to share our CI/CD v2 improvements, product stability work, and what we are prioritizing next across reliability and product roadmap.
{/* truncate */}
## CI/CD v2 improvements
Our CI/CD v2 work is centered around four goals:
1. **Limit** what each package can access
2. **Reduce** the number of sensitive environment variables
3. **Avoid** compromised packages
4. **Reduce the risk of** release tampering
#### New architecture: isolated environments
We have begun moving to isolated environments for distinct CI/CD stages to reduce the chance that a single compromised step can inherit broad access across the entire pipeline.
<Image
img={require('../../img/april_townhall_isolated_environments.png')}
style={{width: '900px', height: 'auto', display: 'block'}}
/>
#### Current rollout status
These changes are deployed in our current release workflow. [See here](https://github.com/BerriAI/litellm/tags)
#### Independently verify releases
A key part of CI/CD v2 is supporting independent verification of release artifacts using our published verification process, while reducing reliance on any single credential or release path.
[**Learn more about how to verify releases**](https://docs.litellm.ai/docs/proxy/docker_image_security)
<Image
img={require('../../img/verify_releases.png')}
style={{width: '900px', height: 'auto', display: 'block'}}
/>
## Stability improvements
### SDLC improvements
This month, we're focusing on process stability improvements around:
- Improving main-branch stability
- Mapping UI QA to built Docker images for 1:1 environment parity
- Consistent release tags across PyPI and Docker
- Fixing release notes publication
#### Improving main-branch stability
We're introducing a staging-gated flow:
<Image
img={require('../../img/stable_main.png')}
style={{width: '900px', height: 'auto', display: 'block'}}
/>
- Only an internal staging branch can push to `main`.
- PRs to that staging branch must pass CircleCI LLM API testing.
- Collision handling happens on staging, which is designed to reduce unstable changes reaching `main`.
#### UI QA in Docker environment
Moving forward, all UI QA will be performed in the built Docker image that users run.
Previously, some UI QA paths were run in local environments that did not fully replicate Docker runtime conditions.
That contributed to release-specific issues, including MCP registration problems in `v1.82.3`.
#### Consistent release tags
Today we publish releases for multiple scenarios:
- Dev (Built of a PR for a customer-specific scenario)
- Nightly (Passes all CI/CD checks)
- Release Candidate (Passes all CI/CD checks + manual UI QA)
- Stable (intended to pass all CI/CD checks + manual UI QA + 7 days of production testing)
We are targeting a consistent naming convention across PyPI and Docker by the end of April.
#### Release notes
CI/CD v2 changes moved release notes to a manual path. This is a temporary solution while we investigate a better automated workflow. We are targeting a more consistent process by the end of April.
### Product stability improvements
#### Stable Prisma migrations
Today, we have observed several migration failure classes:
- Migration not applied
- Migration marked applied but incomplete
- Migration not applied due to non-root image issues
We're prioritizing this work this month and have assigned an engineering owner to the effort. Our target is to resolve these error classes by the end of April.
#### UI type safety
Another area of focus is improving the stability of the UI. Today, one cause of errors is that the UI maintains its own assumptions about backend API types. This can lead to issues when backend responses differ from UI assumptions.
We aim to move to having the UI and Backend be in sync with each other, and are exploring OpenAPI-driven mapping to achieve this.
## Product roadmap
### Our Assumptions
Over the next few years, we expect:
- Companies will give employees more AI tools.
- More AI agents will move into production workflows across HR, finance, support, and operations.
### Our Inferences
#### Near-term
- AI spend will increase.
- Uptime and latency will become even more important.
- More AI resources (skills, CLIs, and related assets) will require governance.
- Agent and MCP usage patterns will require deeper controls.
- Broader developer adoption will increase the need for simpler, more discoverable tooling.
#### Long-term
- We expect many organizations to treat agent auditability (how decisions were made across LLM + MCP + sub-agent inputs/outputs) as a compliance expectation.
- Permission management will get more complex as user-agent interaction chains deepen.
Roadmap timelines in this post are targets and may evolve based on validation and user feedback.
## April investments
### Reliability
- Increase uptime for 10k+ RPS scenarios.
- Investigate latency overhead for long-running Claude Code requests.
### Feature reliability
- Polish MCP authentication.
- Better understand how teams are using agents through LiteLLM.
### Governance
- Launch Skills as a first-class citizen in LiteLLM.
## Q&A
Thank you again for all the questions and direct feedback. We will keep sharing concrete progress updates as these efforts ship.
## Hiring
We are actively hiring across several roles, please apply [here](https://jobs.ashbyhq.com/litellm) if you're interested!

2
docs/my-website/blog/authors.yml
View file

@ -24,7 +24,7 @@ ishaan:
# Alias for typo in name
ishaan-alt:
name: Ishaan Jaff
name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg

159
docs/my-website/blog/redis_circuit_breaker/diagrams.js Normal file
View file

@ -0,0 +1,159 @@
import React from 'react';
const s = {
fig: {margin: '2.5rem 0', fontFamily: 'inherit'},
box: {borderRadius: 12, border: '1px solid #e5e7eb', background: '#fff', padding: '2rem 2.5rem'},
label: {fontSize: 11, fontWeight: 700, textTransform: 'uppercase', letterSpacing: '0.12em', color: '#9ca3af', textAlign: 'center', marginBottom: '1.5rem'},
caption: {textAlign: 'center', fontSize: 12, color: '#9ca3af', marginTop: 12},
node: (border='#d1d5db', bg='#f9fafb') => ({
border: `1px solid ${border}`, borderRadius: 6, padding: '8px 20px',
fontSize: 13, background: bg, display: 'inline-block',
}),
arrow: {display: 'flex', flexDirection: 'column', alignItems: 'center'},
};
const SmallArrow = ({color='#9ca3af'}) => (
<svg width="2" height="28" style={{display:'block'}}>
<line x1="1" y1="0" x2="1" y2="22" stroke={color} strokeWidth="1.5"/>
<polygon points="1,28 -2,21 4,21" fill={color}/>
</svg>
);
export function CascadeFailure() {
return (
<figure style={s.fig}>
<div style={s.box}>
<p style={s.label}>Without circuit breaker — cascade failure</p>
<div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:0}}>
<div style={s.node()}>LiteLLM Pod (×100)</div>
<SmallArrow />
<div style={s.node()}>Rate limit / cache check</div>
<div style={{position:'relative', display:'flex', flexDirection:'column', alignItems:'center'}}>
<SmallArrow color="#f87171"/>
<span style={{position:'absolute', left:8, top:4, fontSize:11, color:'#f87171', whiteSpace:'nowrap'}}>hangs 30s per request</span>
</div>
<div style={s.node('#fca5a5','#fef2f2')}><span style={{color:'#b91c1c', fontWeight:600}}>Redis — degraded, timing out</span></div>
<SmallArrow color="#fb923c"/>
<div style={s.node('#fdba74','#fff7ed')}><span style={{color:'#c2410c', fontWeight:600}}>Postgres — 100× normal read load</span></div>
<SmallArrow />
<div style={{...s.node('#111827','#111827'), color:'#fff', fontWeight:600}}>Total outage — gateway down</div>
</div>
</div>
<figcaption style={s.caption}>Slow Redis → every auth check times out → database overwhelmed → full cascade</figcaption>
</figure>
);
}
export function CircuitBreakerStates() {
const circle = (border, color, label, sub) => (
<div style={{display:'flex', flexDirection:'column', alignItems:'center', width: 140}}>
<div style={{width:88, height:88, borderRadius:'50%', border:`2px solid ${border}`, background:'#fff', display:'flex', flexDirection:'column', alignItems:'center', justifyContent:'center'}}>
<span style={{fontSize:11, fontWeight:700, color, letterSpacing:'0.06em'}}>{label}</span>
<span style={{fontSize:10, color:'#9ca3af', marginTop:2}}>{sub}</span>
</div>
<p style={{fontSize:11, color:'#6b7280', textAlign:'center', marginTop:10, lineHeight:1.5}}>{'\u00a0'}</p>
</div>
);
const arrow = (label) => (
<div style={{display:'flex', flexDirection:'column', alignItems:'center', marginTop:36, marginLeft:4, marginRight:4}}>
<span style={{fontSize:10, color:'#6b7280', marginBottom:4}}>{label}</span>
<div style={{display:'flex', alignItems:'center'}}>
<div style={{height:1, width:48, background:'#9ca3af'}}/>
<svg width="8" height="8" style={{marginLeft:-1}}><polygon points="0,0 8,4 0,8" fill="#6b7280"/></svg>
</div>
</div>
);
return (
<figure style={s.fig}>
<div style={s.box}>
<p style={s.label}>Circuit breaker state machine</p>
<div style={{display:'flex', justifyContent:'center', alignItems:'flex-start'}}>
{circle('#1f2937','#111827','CLOSED','normal')}
{arrow('5 failures')}
{circle('#f87171','#dc2626','OPEN','fast-fail')}
{arrow('60s timeout')}
{circle('#fbbf24','#b45309','HALF-OPEN','probing')}
</div>
<div style={{display:'flex', justifyContent:'center', gap:32, marginTop:24}}>
<div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:4}}>
<div style={{display:'flex', alignItems:'center', gap:4}}>
<svg width="8" height="8"><polygon points="8,0 0,4 8,8" fill="#16a34a"/></svg>
<div style={{height:1, width:100, background:'#16a34a'}}/>
</div>
<span style={{fontSize:10, color:'#16a34a'}}>probe success → CLOSED</span>
</div>
<div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:4}}>
<div style={{display:'flex', alignItems:'center', gap:4}}>
<svg width="8" height="8"><polygon points="8,0 0,4 8,8" fill="#ef4444"/></svg>
<div style={{height:1, width:100, borderTop:'2px dashed #f87171'}}/>
</div>
<span style={{fontSize:10, color:'#ef4444'}}>probe failure → OPEN again</span>
</div>
</div>
</div>
</figure>
);
}
export function CircuitBreakerFlow() {
return (
<figure style={s.fig}>
<div style={s.box}>
<p style={s.label}>With circuit breaker — graceful degradation</p>
<div style={{display:'flex', flexDirection:'column', alignItems:'center'}}>
<div style={s.node()}>Incoming request</div>
<SmallArrow />
<div style={{...s.node('#111827'), border:'2px solid #111827', fontWeight:600}}>Circuit Breaker</div>
<div style={{display:'flex', gap:80, marginTop:20, alignItems:'flex-start'}}>
<div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:8}}>
<SmallArrow />
<span style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.06em', color:'#6b7280', border:'1px solid #e5e7eb', borderRadius:4, padding:'2px 8px'}}>Closed</span>
<div style={{...s.node(), textAlign:'center', fontSize:13}}>Redis call<br/><span style={{fontSize:11, color:'#9ca3af'}}>normal latency</span></div>
</div>
<div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:8}}>
<SmallArrow />
<span style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.06em', color:'#ef4444', border:'1px solid #fca5a5', borderRadius:4, padding:'2px 8px'}}>Open</span>
<div style={{...s.node('#fca5a5'), textAlign:'center', fontSize:13}}>Fast-fail — 0ms<br/><span style={{fontSize:11, color:'#9ca3af'}}>no network call</span></div>
<SmallArrow />
<div style={{...s.node(), textAlign:'center', fontSize:13}}>DB fallback<br/><span style={{fontSize:11, color:'#9ca3af'}}>bounded load</span></div>
</div>
</div>
<div style={{...s.node('#111827','#111827'), color:'#fff', fontWeight:600, marginTop:24}}>Request completes — gateway stays up</div>
</div>
</div>
<figcaption style={s.caption}>Redis down → circuit opens → 0ms rejection → DB absorbs bounded fallback traffic</figcaption>
</figure>
);
}
export function IncidentTimeline() {
const row = (color, text) => (
<div style={{display:'flex', alignItems:'flex-start', gap:10, marginBottom:12}}>
<div style={{marginTop:5, width:6, height:6, borderRadius:'50%', background:color, flexShrink:0}}/>
<p style={{fontSize:13, color:'#4b5563', margin:0, lineHeight:1.5}}>{text}</p>
</div>
);
return (
<figure style={s.fig}>
<div style={s.box}>
<p style={s.label}>Redis degrades — before vs. after</p>
<div style={{display:'grid', gridTemplateColumns:'1fr 1fr', gap:20}}>
<div style={{border:'1px solid #e5e7eb', borderRadius:8, padding:20}}>
<p style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.1em', color:'#9ca3af', marginBottom:16}}>Without circuit breaker</p>
{row('#f87171','All 100 pods hang for 30s on each auth check')}
{row('#f87171','Threadpools fill up, requests queue')}
{row('#f87171','100× simultaneous DB fallbacks overwhelm Postgres')}
{row('#f87171','Requires manual intervention to recover')}
</div>
<div style={{border:'1px solid #111827', borderRadius:8, padding:20}}>
<p style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.1em', color:'#9ca3af', marginBottom:16}}>With circuit breaker</p>
{row('#111827','Circuit opens after 5 failures — 0ms fast-fail')}
{row('#111827','Auth falls back to DB — bounded, not 100× load')}
{row('#111827','Cache miss rate temporarily elevated — gateway stays up')}
{row('#111827','Auto-recovers when Redis comes back — no intervention needed')}
</div>
</div>
</div>
</figure>
);
}

141
docs/my-website/blog/redis_circuit_breaker/index.md Normal file
View file

@ -0,0 +1,141 @@
---
slug: redis-circuit-breaker
title: "Making the AI Gateway Resilient to Redis Failures"
date: 2026-04-11T09:00:00
authors:
- ishaan
description: "How LiteLLM's production AI Gateway handles Redis degradation at scale without cascading failures — circuit breaker pattern, 0ms fast-fail, automatic recovery."
tags: [reliability, redis, infrastructure, engineering, ai-gateway]
hide_table_of_contents: true
---
import { CascadeFailure, CircuitBreakerStates, CircuitBreakerFlow, IncidentTimeline } from './diagrams';
*Last Updated: April 2026*
Enterprise AI Gateway deployments put Redis in the hot path for nearly every request: rate limiting, cache lookups, spend tracking. When Redis is healthy, the latency contribution is single-digit milliseconds — invisible to end users. When it degrades, a production AI Gateway needs to stay up regardless.
Running LiteLLM at scale across 100+ pods means designing for failure modes before they appear. The easy case is Redis going fully down: fail fast, fall through to the database, continue serving requests. The hard case — the one that takes down gateways — is a *slow* Redis: still accepting connections, still responding, but timing out after 20-30 seconds per operation.
{/* truncate */}
## Why slow Redis is harder than a full outage
<CascadeFailure />
With 100 pods each hanging 30 seconds on every auth check, threadpools fill up and requests queue. By the time Redis times out and falls through to Postgres, the database receives 100× its normal load from simultaneous fallbacks. A slow Redis becomes a database outage becomes a full gateway outage. A production-grade AI Gateway cannot allow one degraded dependency to cascade into total failure.
## The fix: circuit breaker pattern
The circuit breaker pattern tracks consecutive failures and cuts off the unhealthy dependency before it cascades. Instead of hanging 30 seconds on each Redis call, the circuit opens after 5 consecutive failures and fast-fails at 0ms — no network call, no wait.
<CircuitBreakerStates />
Three states:
- **CLOSED** — normal. All Redis calls pass through.
- **OPEN** — Redis is unhealthy. Every call fast-fails instantly. Requests continue with degraded-but-functional behavior: auth and rate limiting fall back to the database.
- **HALF-OPEN** — after 60 seconds, one probe request tests recovery. Success closes the circuit; failure resets the timer.
This is how a reliable AI Gateway handles infrastructure degradation: stay up, degrade gracefully, recover automatically.
## How requests flow through the AI Gateway
<CircuitBreakerFlow />
When the circuit is open, the gateway does not stall. Auth checks fall back to Postgres — slower, but bounded. The database absorbs the load because it receives *some* requests via DB fallback, not *all* 100 pods simultaneously dumping their queued requests after a 30-second timeout.
The difference between a resilient AI Gateway and a fragile one: controlled degradation vs. uncontrolled cascade.
## The implementation
```python
class RedisCircuitBreaker:
def __init__(self, failure_threshold: int, recovery_timeout: int):
self.failure_threshold = failure_threshold # default: 5
self.recovery_timeout = recovery_timeout # default: 60s
self._failure_count = 0
self._state = self.CLOSED
def is_open(self) -> bool:
if self._state == self.OPEN:
if time.time() - self._opened_at > self.recovery_timeout:
self._state = self.HALF_OPEN
return False # this caller is the recovery probe
return True # fast-fail
return False
def record_failure(self):
self._failure_count += 1
self._opened_at = time.time()
if self._failure_count >= self.failure_threshold:
self._state = self.OPEN # open the circuit
def record_success(self):
self._failure_count = 0
self._state = self.CLOSED # Redis recovered
```
Every async Redis operation goes through a decorator that checks the breaker before touching the network. When open, it raises immediately:
```python
@_redis_circuit_breaker_guard
async def async_get_cache(self, key: str):
...
```
The decorator handles all bookkeeping — success resets nothing, failures increment the counter, exceptions trigger `record_failure()`. The caller sees a clean exception and falls through to its normal non-Redis path. No changes required in calling code.
## AI Gateway resilience in production
<IncidentTimeline />
Redis degradation events no longer cascade in production. The observable symptom during a Redis slowdown is a temporary bump in cache miss rate — the right failure mode for a resilient AI Gateway. Auth still works. Rate limiting still works. Spend tracking still works, at slightly higher DB cost. Recovery is fully automatic when Redis comes back.
```bash
# configure via environment variables
REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5 # failures before opening
REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT=60 # seconds before probe
```
The circuit breaker ships on by default in all LiteLLM versions since `v1.82.0`. No configuration needed for most deployments.
## Key Takeaways
- A slow Redis is more dangerous than a downed one: 30-second timeouts across 100+ pods overwhelm Postgres at 100× normal load
- LiteLLM's AI Gateway uses a circuit breaker that fast-fails Redis calls at 0ms after 5 consecutive failures
- Three states: CLOSED (normal), OPEN (fast-fail + DB fallback), HALF-OPEN (probe recovery)
- Auth, rate limiting, and spend tracking continue working during Redis outages
- Resilient, production-grade behavior — enabled by default since `v1.82.0`, no configuration required
---
### Frequently Asked Questions
### Does the circuit breaker affect normal Redis performance?
No. When Redis is healthy (circuit CLOSED), every call passes through with zero overhead. The breaker only activates after 5 consecutive failures — transparent under normal conditions.
### What happens to rate limiting when the circuit is open?
Rate limiting falls back to Postgres with bounded load. Limits remain enforced at slightly higher DB cost until Redis recovers and the circuit closes automatically.
### How is this different from basic Redis retry logic?
Retry logic still waits for each timeout (30s × retries). The circuit breaker cuts the connection immediately at 0ms after the failure threshold, preventing threadpool exhaustion across all pods simultaneously. Retries make slow-Redis worse; the circuit breaker contains it.
### Is this available in LiteLLM OSS?
Yes. The circuit breaker ships in LiteLLM OSS (Apache 2.0) by default since `v1.82.0`. [LiteLLM Enterprise](https://litellm.ai/enterprise) adds SSO/SCIM, air-gapped deployment, 24/7 SLA support, and advanced guardrails on top of the OSS foundation.
---
## Conclusion
Redis resilience is one layer of what makes LiteLLM a production-grade, reliable AI Gateway at scale. The circuit breaker pattern ensures infrastructure degradation stays contained — the right failure mode is a temporary cache miss rate bump, not a full outage. This is how AI Gateway infrastructure should behave under pressure: degrade gracefully, recover automatically, keep serving traffic. For teams with strict uptime and compliance requirements, [LiteLLM Enterprise](https://litellm.ai/enterprise) provides the additional controls needed for regulated production environments.
## Recommended Reading
- [LiteLLM AI Gateway — full feature overview](https://docs.litellm.ai/docs/simple_proxy)
- [Load balancing and routing across 100+ LLM providers](https://docs.litellm.ai/docs/routing)
- [Spend tracking and budget controls](https://docs.litellm.ai/docs/proxy/cost_tracking)

94
docs/my-website/docs/benchmarks.md
View file

@ -5,6 +5,55 @@ import Image from '@theme/IdealImage';
Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.
LiteLLM Gateway has **8ms P95 latency** at 1k RPS (See benchmarks [here](#4-instances))
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs:
- 4 CPU
- 8GB RAM
## Configuration
- Database: PostgreSQL
- Redis: Not used
### 2 Instance LiteLLM Proxy
In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.
#### Performance Metrics
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 |
| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 |
| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 |
<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->
<!-- ## **Horizontal Scaling - 10K RPS**
<Image img={require('../img/instances_vs_rps.png')} /> -->
### 4 Instances
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 |
| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 |
| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 |
#### Key Findings
- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.
## Setting Up Benchmarking with Network Mock
The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider.
@ -41,6 +90,8 @@ litellm --config benchmark_config.yaml --port 4000 --num_workers 8
python scripts/benchmark_mock.py --requests 2000 --max-concurrent 200 --runs 3
```
Get the benchmarking script [here](https://github.com/BerriAI/litellm/blob/main/scripts/benchmark_mock.py)
This measures pure proxy overhead on the hot path without any network latency to a real or fake provider.
## Setting Up a Fake OpenAI Endpoint
@ -61,38 +112,6 @@ model_list:
api_key: "test"
```
### 2 Instance LiteLLM Proxy
In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.
#### Performance Metrics
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 |
| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 |
| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 |
<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->
<!-- ## **Horizontal Scaling - 10K RPS**
<Image img={require('../img/instances_vs_rps.png')} /> -->
### 4 Instances
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 |
| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 |
| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 |
#### Key Findings
- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.
## `/realtime` API Benchmarks
End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake realtime endpoint.
@ -115,17 +134,6 @@ End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake
| **System** | 4 vCPUs, 8 GB RAM, 4 workers, 4 instances |
| **Database** | PostgreSQL (Redis unused) |
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs:
- 4 CPU
- 8GB RAM
## Configuration
- Database: PostgreSQL
- Redis: Not used
## Infrastructure Recommendations

422
docs/my-website/docs/completion/anthropic_advisor_tool.md Normal file
View file

@ -0,0 +1,422 @@
# Advisor Tool
Pair a faster executor model with a higher-intelligence advisor model that provides strategic guidance mid-generation.
The advisor tool lets a fast, lower-cost executor model (Sonnet or Haiku) consult a high-intelligence advisor model (Opus 4.6) mid-generation. The advisor reads the full conversation and produces a plan or course correction — typically 400–700 text tokens — and the executor continues with the task.
This pattern is well-suited for long-horizon agentic workloads (coding agents, computer use, multi-step research) where most turns are mechanical but having an excellent plan is crucial. You get close to advisor-solo quality while the bulk of token generation happens at executor-model rates.
:::info Beta
The advisor tool is in beta. Include `anthropic-beta: advisor-tool-2026-03-01` in your requests — LiteLLM adds this automatically when it detects the advisor tool in your `tools` array.
:::
## Supported Providers
| Provider | Chat Completions API | Messages API |
|----------|---------------------|--------------|
| **Anthropic API** | ✅ | ✅ |
| **Azure Anthropic** | ❌ (coming soon) | ❌ (coming soon) |
| **Google Cloud Vertex AI** | ❌ (coming soon) | ❌ (coming soon) |
| **Amazon Bedrock** | ❌ (coming soon) | ❌ (coming soon) |
## Model Compatibility
The executor and advisor models must form a valid pair. Currently the only supported advisor model is `claude-opus-4-6`.
| Executor | Advisor |
|----------|---------|
| `claude-haiku-4-5-20251001` | `claude-opus-4-6` |
| `claude-sonnet-4-6` | `claude-opus-4-6` |
| `claude-opus-4-6` | `claude-opus-4-6` |
---
## Chat Completions API
### SDK Usage
#### Basic Example
```python showLineNumbers title="Advisor Tool — litellm.completion()"
import litellm
response = litellm.completion(
model="anthropic/claude-sonnet-4-6",
messages=[
{"role": "user", "content": "Build a concurrent worker pool in Go with graceful shutdown."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
],
max_tokens=4096,
)
print(response.choices[0].message.content)
```
#### With Optional Parameters
```python showLineNumbers title="Advisor Tool with max_uses and caching"
import litellm
response = litellm.completion(
model="anthropic/claude-sonnet-4-6",
messages=[
{"role": "user", "content": "Build a REST API with authentication in Python."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
"max_uses": 3, # cap advisor calls per request
"caching": {"type": "ephemeral", "ttl": "5m"}, # enable for 3+ calls per conversation
}
],
max_tokens=4096,
)
```
#### Streaming
```python showLineNumbers title="Streaming with Advisor Tool"
import litellm
response = litellm.completion(
model="anthropic/claude-sonnet-4-6",
messages=[
{"role": "user", "content": "Implement a distributed rate limiter."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
],
max_tokens=4096,
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
:::note Streaming behavior
The advisor sub-inference does not stream. The executor's stream pauses while the advisor runs, then the full advisor result arrives in a single event. Executor output resumes streaming afterward.
:::
#### Multi-Turn Conversation
```python showLineNumbers title="Multi-Turn with Advisor Tool"
import litellm
tools = [
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
]
messages = [
{"role": "user", "content": "Build a concurrent worker pool in Go with graceful shutdown."}
]
response = litellm.completion(
model="anthropic/claude-sonnet-4-6",
messages=messages,
tools=tools,
max_tokens=4096,
)
# Append the full response (includes server_tool_use + advisor_tool_result blocks)
messages.append({"role": "assistant", "content": response.choices[0].message.content})
# Continue the conversation — keep the same tools array
messages.append({"role": "user", "content": "Now add a max-in-flight limit of 10."})
response2 = litellm.completion(
model="anthropic/claude-sonnet-4-6",
messages=messages,
tools=tools,
max_tokens=4096,
)
```
:::tip Auto-strip on follow-up turns
LiteLLM automatically strips `advisor_tool_result` blocks from message history when the advisor tool is not present in the current request. This prevents the Anthropic 400 error that would otherwise occur.
:::
### AI Gateway Usage
#### Proxy Configuration
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_API_KEY
```
#### Client Request via Proxy
```python showLineNumbers title="Advisor Tool via AI Gateway"
from openai import OpenAI
client = OpenAI(
api_key="your-litellm-proxy-key",
base_url="http://0.0.0.0:4000/v1"
)
response = client.chat.completions.create(
model="claude-sonnet",
messages=[
{"role": "user", "content": "Implement a distributed rate limiter in Python."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
],
max_tokens=4096,
)
```
---
## Messages API
### SDK Usage
#### Basic Example
```python showLineNumbers title="Advisor Tool — litellm.anthropic.messages"
import asyncio
import litellm
async def main():
response = await litellm.anthropic.messages.acreate(
model="anthropic/claude-sonnet-4-6",
messages=[
{"role": "user", "content": "Build a concurrent worker pool in Go with graceful shutdown."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
],
max_tokens=4096,
)
print(response)
asyncio.run(main())
```
#### Streaming
```python showLineNumbers title="Messages API Streaming with Advisor Tool"
import asyncio
import json
import litellm
async def main():
response = await litellm.anthropic.messages.acreate(
model="anthropic/claude-sonnet-4-6",
messages=[
{"role": "user", "content": "Implement a distributed rate limiter."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
],
max_tokens=4096,
stream=True,
)
async for chunk in response:
if isinstance(chunk, bytes):
for line in chunk.decode("utf-8").split("\n"):
if line.startswith("data: "):
try:
print(json.loads(line[6:]))
except json.JSONDecodeError:
pass
asyncio.run(main())
```
### AI Gateway Usage
#### Proxy Configuration
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_API_KEY
```
#### Client Request via Proxy (Anthropic SDK)
```python showLineNumbers title="Advisor Tool via AI Gateway (Anthropic SDK)"
import anthropic
client = anthropic.Anthropic(
api_key="your-litellm-proxy-key",
base_url="http://0.0.0.0:4000"
)
response = client.beta.messages.create(
model="claude-sonnet",
max_tokens=4096,
betas=["advisor-tool-2026-03-01"],
messages=[
{"role": "user", "content": "Build a concurrent worker pool in Go with graceful shutdown."}
],
tools=[
{
"type": "advisor_20260301",
"name": "advisor",
"model": "claude-opus-4-6",
}
],
)
print(response)
```
---
## Response Structure
A successful advisor call returns `server_tool_use` and `advisor_tool_result` blocks in the assistant content:
```json title="Response with advisor blocks"
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "Let me consult the advisor on this."
},
{
"type": "server_tool_use",
"id": "srvtoolu_abc123",
"name": "advisor",
"input": {}
},
{
"type": "advisor_tool_result",
"tool_use_id": "srvtoolu_abc123",
"content": {
"type": "advisor_result",
"text": "Use a channel-based coordination pattern. The tricky part is draining in-flight work during shutdown: close the input channel first, then wait on a WaitGroup..."
}
},
{
"type": "text",
"text": "Here's the implementation using a channel-based coordination pattern..."
}
]
}
```
Pass the full assistant content, including advisor blocks, back on subsequent turns. LiteLLM handles this automatically through `provider_specific_fields`.
---
## Cost Control
Advisor calls run as a separate sub-inference billed at the advisor model's rates. Usage is reported in `usage.iterations[]`:
```json title="Usage with advisor sub-inference"
{
"usage": {
"input_tokens": 412,
"output_tokens": 531,
"iterations": [
{
"type": "message",
"input_tokens": 412,
"output_tokens": 89
},
{
"type": "advisor_message",
"model": "claude-opus-4-6",
"input_tokens": 823,
"output_tokens": 1612
},
{
"type": "message",
"input_tokens": 1348,
"output_tokens": 442
}
]
}
}
```
Top-level `usage` reflects executor tokens only. Advisor tokens appear in `iterations` entries with `type: "advisor_message"` and are billed at Opus rates.
**Tips:**
- Enable `caching` on the tool definition only when you expect 3+ advisor calls per conversation; it costs more than it saves below that threshold.
- Use `max_uses` to cap advisor calls per request. Once reached, the executor continues without further advice.
- For conversation-level caps, count advisor calls client-side. When you reach your limit, remove the advisor tool from `tools`.
---
## Recommended System Prompt
For coding and agent tasks, Anthropic recommends prepending these blocks to your system prompt for consistent advisor timing and optimal cost/quality:
```text title="Timing guidance (prepend to system prompt)"
You have access to an `advisor` tool backed by a stronger reviewer model. It takes NO parameters — when you call advisor(), your entire conversation history is automatically forwarded. They see the task, every tool call you've made, every result you've seen.
Call advisor BEFORE substantive work — before writing, before committing to an interpretation, before building on an assumption. If the task requires orientation first (finding files, fetching a source, seeing what's there), do that, then call advisor. Orientation is not substantive work. Writing, editing, and declaring an answer are.
Also call advisor:
- When you believe the task is complete. BEFORE this call, make your deliverable durable: write the file, save the result, commit the change.
- When stuck — errors recurring, approach not converging, results that don't fit.
- When considering a change of approach.
On tasks longer than a few steps, call advisor at least once before committing to an approach and once before declaring done. On short reactive tasks where the next action is dictated by tool output you just read, you don't need to keep calling.
```
```text title="Advice weight guidance (add after timing block)"
Give the advice serious weight. If you follow a step and it fails empirically, or you have primary-source evidence that contradicts a specific claim, adapt. A passing self-test is not evidence the advice is wrong.
If you've already retrieved data pointing one way and the advisor points another: don't silently switch. Surface the conflict in one more advisor call — "I found X, you suggest Y, which constraint breaks the tie?"
```
To reduce advisor output length by 35–45% without losing quality, add:
```text title="Cost reduction (optional, add before timing block)"
The advisor should respond in under 100 words and use enumerated steps, not explanations.
```
---
## Additional Resources
- [Anthropic Advisor Tool Documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/advisor-tool)
- [LiteLLM Tool Calling Guide](https://docs.litellm.ai/docs/completion/function_call)

16
docs/my-website/docs/providers/gemini.md
View file

@ -65,14 +65,13 @@ response = completion(
- modalities
- reasoning_content
- audio (for TTS models only)
- service_tier
**Anthropic Params**
- thinking (used to set max budget tokens across anthropic/gemini models)
[**See Updated List**](https://github.com/BerriAI/litellm/blob/main/litellm/llms/gemini/chat/transformation.py#L70)
## Usage - Thinking / `reasoning_content`
LiteLLM translates OpenAI's `reasoning_effort` to Gemini's `thinking` parameter. [Code](https://github.com/BerriAI/litellm/blob/620664921902d7a9bfb29897a7b27c1a7ef4ddfb/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py#L362)
@ -298,6 +297,19 @@ curl http://0.0.0.0:4000/v1/chat/completions \
## Usage - `service_tier`
LiteLLM propagates OpenAI's `service_tier` parameter to Gemini, and also extracts it from the response headers (`x-gemini-service-tier`) into `model_response.service_tier`.
| OpenAI `service_tier` | Gemini `service_tier` | Notes |
| --------------------- | --------------------- | ----- |
| `"auto"` | `"priority"` | LiteLLM maps OpenAI's `"auto"` to Gemini's `"priority"` tier, as `priority` will fall back on Gemini. |
| `"flex"` | `"flex"` | Direct mapping. |
| `"priority"` | `"priority"` | Direct mapping. |
| `"default"` | `"standard"` | LiteLLM maps `"default"` to `"standard"`. |
| Any other value | Passed as-is (lowercased) | Values are case-insensitive and normalized to lowercase. |
On the response, LiteLLM maps `"standard"` back to `"default"` for the Gemini API.
## Text-to-Speech (TTS) Audio Output

267
docs/my-website/docs/providers/sap.md
View file

@ -55,24 +55,33 @@ pip install litellm
```
### Step 2: Set Your Credentials
Choose **one** of these authentication methods:
> **Breaking change**: credential resolution is "first-source-wins"
>
> Credential resolution no longer merges individual fields across sources.
>
> Resolution order is:
`kwargs` → `service key` → `env (AICORE_*)` → `config` → `VCAP service`
>
> **Important behavior:** once LiteLLM finds *any* credential value in a source, it takes **all** credentials from that source exclusively (except `resource_group`, which may still be resolved separately).
Choose **one** of these authentication methods:
<Tabs>
<TabItem value="service-key" label="Service Key JSON (Recommended)">
<Tabs>
<TabItem value="service-key" label="Service Key JSON (Recommended)">
The simplest approach - paste your entire service key as a single environment variable.
The simplest approach - paste your entire service key as a single environment variable. The service key must be wrapped in a `credentials` object:
> **Note:** the service key no more needs to be wrapped in a "credentials" key.
```bash
export AICORE_SERVICE_KEY='{
"credentials": {
"clientid": "your-client-id",
"clientsecret": "your-client-secret",
"url": "https://<your-instance>.authentication.sap.hana.ondemand.com",
"serviceurls": {
"AI_API_URL": "https://api.ai.<your-region>.aws.ml.hana.ondemand.com"
}
}
}'
export AICORE_RESOURCE_GROUP="default"
```
@ -220,6 +229,17 @@ model="sap/gemini-2.5-pro"
# Incorrect - missing prefix
model="gpt-4o" # ❌ Won't work
```
3. **Environment variables** - Set the following list of credentials in .env file
<pre>
AICORE_AUTH_URL = "https://* * * .authentication.sap.hana.ondemand.com/oauth/token",
AICORE_CLIENT_ID = " *** ",
AICORE_CLIENT_SECRET = " *** ",
AICORE_RESOURCE_GROUP = " *** ",
AICORE_BASE_URL = "https://api.ai.***.cfapps.sap.hana.ondemand.com/v2"
</pre>
Other credential configuration options are also available. For more information, see the [SAP AI Core Documentation](https://help.sap.com/doc/generative-ai-hub-sdk/CLOUD/en-US/_reference/README_sphynx.html#configuration).
## Usage - LiteLLM Python SDK
### Proxy Usage
@ -506,6 +526,241 @@ response = embedding(
print(response.data[0]["embedding"]) # Vector representation
```
### Additional Modules
The SAP Gen AI Hub includes additional modules for advanced use cases:
- [Grounding](https://help.sap.com/docs/sap-ai-core/generative-ai/grounding-035c455a5a424697b60f4a24b6d791fe?locale=en-US)
- [Translation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US)
- [Data Masking](https://help.sap.com/docs/sap-ai-core/generative-ai/data-masking-d9a54d9ca54b40beacbd24e1663ec3b4?locale=en-US)
- [Content Filtering](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US)
#### Grounding
Grounding is a service designed to handle data-related tasks, such as grounding and retrieval, using vector databases. It provides specialized data retrieval through these databases, grounding the retrieval process with your own external and context-relevant data. Grounding combines generative AI capabilities with the ability to use real-time, precise data to improve decision-making and business operations for specific AI-driven business solutions.
##### Prerequisites
To use the Grounding module in the orchestration pipeline, you need to prepare the knowledge base in advance.
Generative AI hub offers multiple options for users to provide data (prepare a knowledge base):
- For Option 1: Upload the documents to a supported data repository and run the data pipeline to vectorize the documents.
- For Option 2: Provide the chunks of document via Vector API directly.
To use grounding, choose from one of the following options.
Usage example:
```python showLineNumbers title="Grounding Example"
from litellm import completion
grounding_config = {
'type': 'document_grounding_service',
'config': {
'filters': [
{'id': 's3-docs',
'data_repository_type': 'vector',
'search_config': {'max_chunk_count': 2},
'data_repositories': ['012345-6789-0123-4567-890123456789']
}
],
'placeholders': {'input': ['user_query'], 'output': 'grounding_response'},
'metadata_params': ['source', 'webUrl', 'title', 'mimeType', 'fileSuffix']
}
}
response = completion(model="sap/gpt-4o",
messages=[
{"content":"""Facility Solutions Company provides services to luxury residential complexes,
apartments, individual homes, and commercial properties such as office buildings, retail
spaces, industrial facilities, and educational institutions. Customers are encouraged to
reach out with maintenance requests, service deficiencies, follow-ups, or any issues they
need by email.""", "role": "system"},
{"content":"""You are a helpful assistant for any queries for answering questions.
Answer the request by providing relevant answers that fit to the request.
Request: {{ ?user_query }}
Context:{{ ?grounding_response }}""", "role": "user"}
],
placeholder_values={"user_query": "Is there a complaint?"},
grounding=grounding_config
)
print(response.choices[0].message.content)
```
For more information about all available grounding configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/using-grounding-module-e1c4dd100dfb42ab890e1d95f3516187?locale=en-US).
#### Translation
The translation module allows you to translate LLM text prompts into a chosen target language.
```python showLineNumbers title="Translation Example"
from litellm import completion
translation_config = {
'input':
{'type': 'sap_document_translation',
'config':
{'source_language': 'en-US',
'target_language': 'de-DE'}
},
'output':
{'type': 'sap_document_translation',
'config':
{'source_language': 'de-DE',
'target_language': 'fr-FR'}
}
}
response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Hello world!"}],
translation=translation_config)
print(response.choices[0].message.content)
```
For more information about all available translation configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US)
#### Data Masking
The data masking module serves to anonymize or pseudonymize personally identifiable information from the input for selected entities.
```python showLineNumbers title="Data Masking Example"
from litellm import completion, embedding
masking_config = {
'providers':
[
{
'type': 'sap_data_privacy_integration',
'method': 'anonymization',
'entities': [
{'type': 'profile-address'},
{'type': 'profile-email'},
{'type': 'profile-phone'},
{'type': 'profile-person'},
{'type': 'profile-location'}
]
}
]
}
mock_cv = "some text with personal information"
response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Give a one sentence summary of the CV. CV: {{?cv}}?"}],
placeholder_values={"cv": mock_cv},
masking=masking_config)
print(response.choices[0].message.content)
# Data masking module also available for embedding
response = embedding(model="sap/text-embedding-3-small",
input=mock_cv,
masking=masking_config)
print(response.data[0])
```
For more information about all available data masking configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/enhancing-model-consumption-with-data-masking-66ad6f469afc4c2cbaa91a27a33f7b21?locale=en-US)
#### Content Filtering
The content filtering module allows you to filter input and output based on content safety criteria.
The module supports two services:
* Azure Content Safety
* Llama Guard 3
```python showLineNumbers title="Content Filtering Example"
from litellm import completion
filtering_config_azure = {
'input':
{
'filters':
[
{'type': 'azure_content_safety',
'config':
{'hate': 0,
'sexual': 0,
'violence': 0,
'self_harm': 0
}
}
]
},
'output':
{
'filters':
[
{'type': 'azure_content_safety',
'config': {'hate': 0,
'sexual': 0,
'violence': 0,
'self_harm': 0
}
}
]
}
}
response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Hello world!"}],
filtering=filtering_config_azure)
print(response.choices[0].message.content)
# The model responds normally because the content does not violate any safety rules.
try:
response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "I hate you"}],
filtering=filtering_config_azure)
except Exception as e:
print(e)
# The service raises an error:
# "Input Filter: Content filtered due to safety violations. Please modify the prompt and try again."
```
For more information about all available content filtering configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US)
#### List of modules configuration for fallback
SAP GEN AI Hub supports a fallback mechanism for handling errors. This mechanism allows you to specify a list of fallback modules to use in case of errors. The fallback modules should contain all parameters that are required for configuring the request.
Required parameters:
- `model`
- `messages`
Optional parameters:
- `filtering`
- `grounding`
- `translation`
- `masking`
- `tools`
- and any of model's specific parameters.
```python showLineNumbers title="Fallback Example"
from litellm import completion
translation_config = {
'input':
{'type': 'sap_document_translation',
'config':
{'source_language': 'en-US',
'target_language': 'de-DE'}
},
'output':
{'type': 'sap_document_translation',
'config':
{'source_language': 'de-DE',
'target_language': 'fr-FR'}
}
}
response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Hello world!"}],
translation=translation_config,
fallback_sap_modules=[{
"model":"sap/gemini-2.5-flash",
"messages":[{"role": "user", "content": "Hello world!"}],
"translation":translation_config
}])
# In case of error with the first configuration (model gpt-4o), the fallback module is used.
print(response.choices[0].message.content)
```
## Reference
### Supported Parameters

3
docs/my-website/docs/proxy/config_settings.md
View file

@ -197,6 +197,7 @@ router_settings:
| key_generation_settings | object | Restricts who can generate keys. [Further docs](./virtual_keys.md#restricting-key-generation) |
| disable_add_transform_inline_image_block | boolean | For Fireworks AI models - if true, turns off the auto-add of `#transform=inline` to the url of the image_url, if the model is not a vision model. |
| use_chat_completions_url_for_anthropic_messages | boolean | If true, routes OpenAI `/v1/messages` requests through chat/completions instead of the Responses API. Can also be set via env var `LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true`. |
| skip_system_message_in_guardrail | boolean | If true, unified guardrails omit `role: system` from scanned input on **chat completions** and **Anthropic `/v1/messages`** only; the LLM still receives full messages. Per-guardrail override: `litellm_params.skip_system_message_in_guardrail` on each guardrail. [Guardrails quick start](./guardrails/quick_start#skip-system-messages-in-guardrail-evaluation) |
| disable_hf_tokenizer_download | boolean | If true, it defaults to using the openai tokenizer for all models (including huggingface models). |
| enable_json_schema_validation | boolean | If true, enables json schema validation for all requests. |
| enable_key_alias_format_validation | boolean | If true, validates `key_alias` format on `/key/generate` and `/key/update`. Must be 2-255 chars, start/end with alphanumeric, only allow `a-zA-Z0-9_-/.@`. Default `false`. |
@ -602,6 +603,8 @@ router_settings:
| MCP_OAUTH2_TOKEN_CACHE_MAX_SIZE | Maximum number of entries in MCP OAuth2 token cache. Default is 200
| MCP_OAUTH2_TOKEN_CACHE_MIN_TTL | Minimum TTL in seconds for MCP OAuth2 token cache. Default is 10
| MCP_OAUTH2_TOKEN_EXPIRY_BUFFER_SECONDS | Seconds to subtract from token expiry when computing cache TTL. Default is 60
| MCP_PER_USER_TOKEN_DEFAULT_TTL | Default TTL in seconds for per-user MCP OAuth tokens stored in Redis. Default is 43200 (12 hours)
| MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS | Seconds to subtract from per-user MCP OAuth token expiry when computing Redis TTL. Default is 60
| DEFAULT_MOCK_RESPONSE_COMPLETION_TOKEN_COUNT | Default token count for mock response completions. Default is 20
| DEFAULT_MOCK_RESPONSE_PROMPT_TOKEN_COUNT | Default token count for mock response prompts. Default is 10
| DEFAULT_MODEL_CREATED_AT_TIME | Default creation timestamp for models. Default is 1677610602

274
docs/my-website/docs/proxy/credential_routing.md Normal file
View file

@ -0,0 +1,274 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Per-Team/Project Credential Routing
Route the same model to different LLM provider endpoints (e.g. different Azure instances) based on which team or project makes the request.
## Overview
In multi-tenant deployments, different teams often need the same model name (e.g., `gpt-4`) to hit different provider endpoints — for example, separate Azure OpenAI instances per business unit for cost isolation, data residency, or rate limit separation.
**Credential routing** lets you configure this in team/project metadata using the existing [credentials table](./ui_credentials.md), without duplicating model definitions or creating separate model groups per team.
```
Hotel Team → gpt-4 → https://hotel-eastus.openai.azure.com/
Flight Team → gpt-4 → https://flight-centralus.openai.azure.com/
```
### Precedence Chain
When a request comes in, the system walks this precedence chain (first match wins):
1. **Clientside credentials** — `api_base`/`api_key` passed in the request body ([docs](./clientside_auth.md))
2. **Project model-specific** — override for this exact model in the project's `model_config`
3. **Project default** — `defaultconfig` in the project's `model_config`
4. **Team model-specific** — override for this exact model in the team's `model_config`
5. **Team default** — `defaultconfig` in the team's `model_config`
6. **Deployment default** — the model's `litellm_params` as configured in `config.yaml`
## Quick Start
### Step 1: Create Credentials
Store your Azure endpoint credentials in the credentials table. You can do this via the [UI](./ui_credentials.md) or API:
```bash showLineNumbers
# Create credential for Hotel team's Azure endpoint
curl -X POST 'http://0.0.0.0:4000/credentials' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"credential_name": "hotel-azure-eastus",
"credential_values": {
"api_base": "https://hotel-eastus.openai.azure.com/",
"api_key": "sk-azure-hotel-key-xxx"
}
}'
```
```bash showLineNumbers
# Create credential for Flight team's Azure endpoint
curl -X POST 'http://0.0.0.0:4000/credentials' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"credential_name": "flight-azure-centralus",
"credential_values": {
"api_base": "https://flight-centralus.openai.azure.com/",
"api_key": "sk-azure-flight-key-xxx"
}
}'
```
### Step 2: Set `model_config` on Teams
Add a `model_config` key to the team's metadata referencing the credential by name:
```bash showLineNumbers
# Hotel team — default Azure endpoint for all models
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"team_id": "hotel-team-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "hotel-azure-eastus"
}
}
}
}
}'
```
```bash showLineNumbers
# Flight team — default Azure endpoint for all models
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"team_id": "flight-team-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "flight-azure-centralus"
}
}
}
}
}'
```
### Step 3: Make Requests
Requests are automatically routed to the correct Azure endpoint based on the API key's team:
```bash showLineNumbers
# Request using Hotel team's API key → routes to hotel-eastus.openai.azure.com
curl http://localhost:4000/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-hotel-team-key' \
-d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
# Request using Flight team's API key → routes to flight-centralus.openai.azure.com
curl http://localhost:4000/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-flight-team-key' \
-d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Per-Model Overrides
You can set different credentials for specific models while keeping a default for everything else:
```bash showLineNumbers
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"team_id": "hotel-team-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "hotel-azure-eastus"
}
},
"gpt-4": {
"azure": {
"litellm_credentials": "hotel-azure-westus"
}
}
}
}
}'
```
With this config:
- `gpt-4` requests → `hotel-azure-westus` credential (model-specific)
- All other models → `hotel-azure-eastus` credential (default)
## Project-Level Overrides
Projects inherit their team's `model_config` but can override at the project level. Project overrides take precedence over team overrides.
```bash showLineNumbers
# Project overrides the team default for all models
curl -X PATCH 'http://0.0.0.0:4000/project/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"project_id": "hotel-rec-app-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "hotel-rec-azure"
}
},
"gpt-4-vision": {
"azure": {
"litellm_credentials": "hotel-rec-vision"
}
}
}
}
}'
```
### Full Example: Hotel Team with Two Projects
**Setup:**
- **Hotel Team**: default `hotel-azure-eastus`, GPT-4 override to `hotel-azure-westus`
- **Hotel Rec App** (project): default `hotel-rec-azure`, GPT-4-Vision override to `hotel-rec-vision`
- **Hotel Review App** (project): no overrides — inherits team config
**Resolution:**
| Request | Resolved Credential | Why |
|---|---|---|
| Hotel Rec App → `gpt-4` | `hotel-rec-azure` | Project default (no project model-specific match for gpt-4) |
| Hotel Rec App → `gpt-4-vision` | `hotel-rec-vision` | Project model-specific |
| Hotel Review App → `gpt-3.5` | `hotel-azure-eastus` | Team default (no project config) |
| Hotel Review App → `gpt-4` | `hotel-azure-westus` | Team model-specific |
## `model_config` Schema
The `model_config` key is a JSON object in team/project `metadata`:
```json
{
"model_config": {
"defaultconfig": {
"<provider>": {
"litellm_credentials": "<credential-name>"
}
},
"<model-name>": {
"<provider>": {
"litellm_credentials": "<credential-name>"
}
}
}
}
```
| Field | Description |
|---|---|
| `defaultconfig` | Fallback credential for any model not explicitly listed |
| `<model-name>` | Model-specific override — must match the LiteLLM model group name |
| `<provider>` | Provider key (e.g. `azure`, `openai`, `bedrock`). When the model name includes a provider prefix (e.g. `azure/gpt-4`), the system prefers the matching provider key |
| `litellm_credentials` | Name of a credential in the [credentials table](./ui_credentials.md) |
### Credential Values
The referenced credential can contain any combination of:
| Key | Description |
|---|---|
| `api_base` | Provider endpoint URL |
| `api_key` | API key for the provider |
| `api_version` | API version (e.g. for Azure) |
Only keys present in the credential are applied. Keys already in the request (e.g. clientside `api_version`) are never overwritten.
## Enabling the Feature
This feature is **disabled by default** and must be explicitly enabled. To enable it:
<Tabs>
<TabItem value="config" label="config.yaml">
```yaml
litellm_settings:
enable_model_config_credential_overrides: true
```
</TabItem>
<TabItem value="env" label="Environment Variable">
```bash
export LITELLM_ENABLE_MODEL_CONFIG_CREDENTIAL_OVERRIDES=true
```
</TabItem>
</Tabs>
:::info
The feature flag must be enabled before `model_config` entries in team/project metadata take effect. Without it, credential routing is completely inert — no metadata is read, no credentials are resolved.
:::
## Related Documentation
- [Adding LLM Credentials](./ui_credentials.md) — Create and manage reusable credentials
- [Project Management](./project_management.md) — Project hierarchy and API
- [Team Budgets](./team_budgets.md) — Team-level budget management
- [Clientside LLM Credentials](./clientside_auth.md) — Passing credentials in the request body
- [Credential Usage Tracking](./credential_usage_tracking.md) — Track spend by credential

2
docs/my-website/docs/proxy/deploy.md
View file

@ -99,7 +99,7 @@ The following checks were performed on each of these signatures:
- The signatures were verified against the specified public key
```
Learn more about LiteLLM's release signing in the [CI/CD v2 announcement](https://docs.litellm.ai/blog/ci-cd-v2-improvements#verify-docker-image-signatures).
Learn more about LiteLLM's release signing in the [CI/CD v2 announcement](https://docs.litellm.ai/blog/ci-cd-v2-improvements#verify-docker-image-signatures). For a complete guide covering all image variants, CI/CD enforcement, and deployment best practices, see the [Docker Image Security Guide](./docker_image_security.md).
### Docker Run

189
docs/my-website/docs/proxy/docker_image_security.md Normal file
View file

@ -0,0 +1,189 @@
# Docker Image Security Guide
LiteLLM signs every Docker image published to GHCR with [cosign](https://docs.sigstore.dev/cosign/overview/) starting from **v1.83.0**. This page covers how to verify signatures, enforce verification in CI/CD, and follow recommended deployment patterns.
## Signed images
All image variants published to `ghcr.io/berriai/` are signed with the same cosign key:
| Image | Description |
|---|---|
| `ghcr.io/berriai/litellm` | Core proxy |
| `ghcr.io/berriai/litellm-database` | Proxy with Postgres dependencies |
| `ghcr.io/berriai/litellm-non_root` | Non-root variant |
| `ghcr.io/berriai/litellm-spend_logs` | Spend-logs sidecar |
The signing key was introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0) and the public key is checked into the repository at [`cosign.pub`](https://github.com/BerriAI/litellm/blob/main/cosign.pub).
:::info Enterprise images
Enterprise images (`litellm-ee`) follow the same signing process. Contact [support@berri.ai](mailto:support@berri.ai) to confirm coverage for your specific enterprise image tag.
:::
## Verify image signatures
Install cosign following the [official instructions](https://docs.sigstore.dev/cosign/system_config/installation/).
### Verify with the pinned commit hash (recommended)
A commit hash is cryptographically immutable, making this the strongest verification method:
```bash
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm:v1.83.0-stable
```
Replace the image reference with any signed variant:
```bash
# litellm-database
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm-database:v1.83.0-stable
# litellm-non_root
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm-non_root:v1.83.0-stable
```
### Verify with a release tag (convenience)
Tags are protected in this repository and resolve to the same key:
```bash
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/v1.83.0-stable/cosign.pub \
ghcr.io/berriai/litellm-database:v1.83.0-stable
```
### Expected output
```
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
```
## Enforce verification in CI/CD
### Kubernetes — Sigstore Policy Controller
The [Sigstore Policy Controller](https://docs.sigstore.dev/policy-controller/overview/) rejects pods whose images fail cosign verification.
1. Install the controller:
```bash
helm repo add sigstore https://sigstore.github.io/helm-charts
helm install policy-controller sigstore/policy-controller \
-n cosign-system --create-namespace
```
2. Create a `ClusterImagePolicy` with the LiteLLM public key:
```yaml
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: litellm-signed-images
spec:
images:
- glob: "ghcr.io/berriai/litellm*"
authorities:
- key:
data: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKi4ivqGpE231OGH50PKbqy1Y1Kkb
POJC8+i2Wko82gBOUCe3M0Vw86H/4rhUhfoYEti4gdJ9wZbYmK0I2EE96g==
-----END PUBLIC KEY-----
```
3. Label the namespace to enable enforcement:
```bash
kubectl label namespace litellm policy.sigstore.dev/include=true
```
Any pod in that namespace using an unsigned `ghcr.io/berriai/litellm*` image will be rejected at admission.
### GCP — Binary Authorization
[Binary Authorization](https://cloud.google.com/binary-authorization/docs) can enforce cosign signatures on Cloud Run and GKE.
1. Create a cosign-based attestor using the LiteLLM public key:
```bash
# Import the public key into a Cloud KMS keyring or use a PGP/PKIX attestor.
# See: https://cloud.google.com/binary-authorization/docs/creating-attestors-console
```
2. Configure a Binary Authorization policy that requires the attestor for `ghcr.io/berriai/litellm*` images.
3. Enable the policy on your Cloud Run service or GKE cluster.
Refer to the [GCP Binary Authorization docs](https://cloud.google.com/binary-authorization/docs/setting-up) for full setup steps.
### AWS — ECS / ECR
AWS does not natively verify cosign signatures at deploy time. Common approaches:
- **CI/CD gate**: Run `cosign verify` in your deployment pipeline before pushing to ECR or updating the ECS task definition. Fail the pipeline if verification fails.
- **OPA/Gatekeeper on EKS**: If running on EKS, use the Sigstore Policy Controller (same as the Kubernetes approach above).
### GitHub Actions gate
Add a verification step before any deployment job:
```yaml
- name: Verify LiteLLM image signature
run: |
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm-database:${{ env.LITELLM_VERSION }}
```
## Recommended deployment patterns
### Pin by digest
Digest pinning guarantees the exact image content regardless of tag mutations:
```yaml
image: ghcr.io/berriai/litellm-database@sha256:<digest>
```
Get the digest after pulling:
```bash
docker inspect --format='{{index .RepoDigests 0}}' \
ghcr.io/berriai/litellm-database:v1.83.0-stable
```
Cosign verification works with digests too:
```bash
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm-database@sha256:<digest>
```
### Use stable release tags
If digest pinning is too rigid for your workflow, use `-stable` release tags (e.g. `v1.83.0-stable`). These are immutable release tags that will not be overwritten.
Avoid `main-latest` or `main-stable` in production — these rolling tags point to the most recent build and can change between deployments.
### Safe upgrade checklist
1. **Verify the new image** — run `cosign verify` against the new release tag or digest.
2. **Test in staging** — deploy the verified image to a non-production environment.
3. **Update your pinned reference** — change the digest or tag in your deployment manifest.
4. **Deploy to production** — roll out using your standard deployment process.
5. **Monitor `/health`** — confirm the proxy is healthy after the upgrade.
## Further reading
- [CI/CD v2 announcement](https://docs.litellm.ai/blog/ci-cd-v2-improvements) — background on LiteLLM's signing infrastructure
- [Docker deployment guide](./deploy.md) — full Docker, Helm, and Terraform setup
- [cosign documentation](https://docs.sigstore.dev/cosign/overview/) — cosign usage and key management
- [Sigstore Policy Controller](https://docs.sigstore.dev/policy-controller/overview/) — Kubernetes admission control

117
docs/my-website/docs/proxy/guardrails/quick_start.md
View file

@ -9,6 +9,7 @@ Setup Prompt Injection Detection, PII Masking on LiteLLM Proxy (AI Gateway)
## 1. Define guardrails on your LiteLLM config.yaml
Set your guardrails under the `guardrails` section
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -82,27 +83,58 @@ For generic guardrail APIs you can also set **static headers** (`headers`: key/v
- `during_call` Run **during** LLM call, on **input** Same as `pre_call` but runs in parallel as LLM call. Response not returned until guardrail check completes
- A list of the above values to run multiple modes, e.g. `mode: [pre_call, post_call]`
### Skip system messages in guardrail evaluation
You can stop **unified** guardrails from scanning `role: system` content while still sending the full `messages` list to the model.
**Global** — in `litellm_settings`:
```yaml
litellm_settings:
skip_system_message_in_guardrail: true
```
**Per guardrail** — under that guardrail’s `litellm_params`: set `skip_system_message_in_guardrail: true` or `false`. If omitted, the global `litellm_settings` value is used; per-guardrail `false` forces system messages to be included even when the global flag is `true`.
**Via LiteLLM UI** — when **creating** or **editing** a guardrail in the LiteLLM Admin Dashboard, set **Skip system messages in guardrail** (under Basic Info on create, or in the edit / guardrail settings flows):
| UI option | Effect |
| ------------------------------------- | -------------------------------------------------------------------------------------- |
| **Use global default** | Uses `litellm_settings.skip_system_message_in_guardrail` from your proxy config |
| **Yes — exclude from guardrail scan** | Sets per-guardrail `skip_system_message_in_guardrail: true` |
| **No — always include in scan** | Sets per-guardrail `skip_system_message_in_guardrail: false` (overrides a global skip) |
<Image
img={require('../../../img/skip_system_message_guardrail_ui.png')}
alt="Create guardrail: Skip system messages in guardrail dropdown with Use global default, Yes exclude from guardrail scan, and No always include in scan"
style={{ width: '100%', maxWidth: '900px', height: 'auto' }}
/>
**Where this applies:** Only the **unified** guardrail path (providers that implement `apply_guardrail` and run through LiteLLM’s message translation layer) on **OpenAI Chat Completions** (`/v1/chat/completions`) and **Anthropic Messages** (`/v1/messages`). Examples include Presidio, Bedrock guardrails, `litellm_content_filter`, OpenAI Moderation, Generic Guardrail API, and custom code guardrails that define `apply_guardrail`.
**Where this does *not* apply:** Guardrails that run only via direct hooks on the raw request (e.g. Lakera v2, Aporia, DynamoAI, Javelin, Lasso, Pangea, Model Armor, Azure Content Safety hooks, Guardrails AI, AIM, tool permission, MCP security). It also does not apply to other routes until those endpoints use the same translation layer (e.g. Responses API, embeddings, speech).
### Load Balancing Guardrails
Need to distribute guardrail requests across multiple accounts or regions? See [Guardrail Load Balancing](./guardrail_load_balancing.md) for details on:
- Load balancing across multiple AWS Bedrock accounts (useful for rate limit management)
- Weighted distribution across guardrail instances
- Multi-region guardrail deployments
## 2. Start LiteLLM Gateway
## 2. Start LiteLLM Gateway
```shell
litellm --config config.yaml --detailed_debug
```
## 3. Test request
## 3. Test request
**[Langchain, OpenAI SDK Usage Examples](../proxy/user_keys#request-format)**
<Tabs>
<TabItem label="Unsuccessful call" value = "not-allowed">
Expect this to fail since since `ishaan@berri.ai` in the request is PII
@ -141,9 +173,9 @@ Expected response on failure
```
</TabItem>
<TabItem label="Successful Call " value = "allowed">
```shell
curl -i http://localhost:4000/v1/chat/completions \
@ -158,10 +190,8 @@ curl -i http://localhost:4000/v1/chat/completions \
}'
```
</TabItem>
</Tabs>
## **Default On Guardrails**
@ -183,7 +213,6 @@ guardrails:
In this request, the guardrail `aporia-pre-guard` will run on every request because `default_on: true` is set.
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
@ -207,6 +236,7 @@ x-litellm-applied-guardrails: aporia-pre-guard
### Guardrail Policies
Need more control? Use [Guardrail Policies](./guardrail_policies.md) to:
- Group guardrails into reusable policies
- Enable/disable guardrails for specific teams, keys, or models
- Inherit from existing policies and override specific guardrails
@ -217,7 +247,6 @@ Need more control? Use [Guardrail Policies](./guardrail_policies.md) to:
Pass `guardrails` to your request body to test it
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
@ -239,7 +268,6 @@ Follow this simple workflow to implement and tune guardrails:
First, check what guardrails are available and their parameters:
Call `/guardrails/list` to view available guardrails and the guardrail info (supported parameters, description, etc)
```shell
@ -271,9 +299,12 @@ Expected response
}
```
>
This config will return the `/guardrails/list` response above. The `guardrail_info` field is optional and you can add any fields under info for consumers of your guardrail
>
```yaml
- guardrail_name: "aporia-post-guard"
litellm_params:
@ -291,9 +322,10 @@ This config will return the `/guardrails/list` response above. The `guardrail_in
type: "boolean"
```
### 2. Apply Guardrails
Add selected guardrails to your chat completion request:
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
@ -322,7 +354,6 @@ curl -i http://localhost:4000/v1/chat/completions \
}'
```
### 4. ✨ Pass Dynamic Parameters to Guardrail
:::info
@ -334,9 +365,8 @@ curl -i http://localhost:4000/v1/chat/completions \
Use this to pass additional parameters to the guardrail API call. e.g. things like success threshold. **[See `guardrails` spec for more details](#spec-guardrails-parameter)**
<Tabs>
<TabItem value="openai" label="OpenAI Python v1.0.0+">
Set `guardrails={"aporia-pre-guard": {"extra_body": {"success_threshold": 0.9}}}` to pass additional parameters to the guardrail
@ -371,10 +401,10 @@ response = client.chat.completions.create(
print(response)
```
</TabItem>
<TabItem value="Curl" label="Curl Request">
```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \
@ -396,11 +426,8 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}
}'
```
</TabItem>
</Tabs>
@ -426,9 +453,6 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g
<Image img={require('../../../img/gd_fail.png')} />
### ✨ Control Guardrails per API Key
:::info
@ -438,12 +462,12 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g
:::
Use this to control what guardrails run per API Key. In this tutorial we only want the following guardrails to run for 1 API Key
- `guardrails`: ["aporia-pre-guard", "aporia-post-guard"]
**Step 1** Create Key with guardrail settings
<Tabs>
<TabItem value="/key/generate" label="/key/generate">
```shell
curl -X POST 'http://0.0.0.0:4000/key/generate' \
@ -454,8 +478,7 @@ curl -X POST 'http://0.0.0.0:4000/key/generate' \
}'
```
</TabItem>
<TabItem value="/key/update" label="/key/update">
```shell
curl --location 'http://0.0.0.0:4000/key/update' \
@ -467,8 +490,7 @@ curl --location 'http://0.0.0.0:4000/key/update' \
}'
```
</TabItem>
</Tabs>
**Step 2** Test it with new key
@ -499,8 +521,7 @@ Run guardrails based on the user-agent header. This is useful for running pre-ca
Both `default` and tag values can be a single mode string or a list of modes.
<Tabs>
<TabItem value="single" label="Single Default Mode">
```yaml
model_list:
@ -522,11 +543,10 @@ guardrails:
default_on: true # run on every request
```
</TabItem>
<TabItem value="multi" label="Multiple Default Modes">
```yaml
model_list:
Per guardrailmodel_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
@ -545,8 +565,7 @@ guardrails:
default_on: true
```
</TabItem>
<TabItem value="tag-list" label="Multiple Tag Modes">
```yaml
model_list:
@ -568,8 +587,6 @@ guardrails:
default_on: true
```
</TabItem>
</Tabs>
### ✨ Model-level Guardrails
@ -580,10 +597,8 @@ guardrails:
:::
This is great for cases when you have an on-prem and hosted model, and just want to run prevent sending PII to the hosted model.
```yaml
model_list:
- model_name: claude-sonnet-4
@ -620,8 +635,7 @@ guardrails:
:::
#### 1. Disable team from modifying guardrails
#### 1. Disable team from modifying guardrails
```bash
curl -X POST 'http://0.0.0.0:4000/team/update' \
@ -633,7 +647,7 @@ curl -X POST 'http://0.0.0.0:4000/team/update' \
}'
```
#### 2. Try to disable guardrails for a call
#### 2. Try to disable guardrails for a call
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
@ -672,8 +686,7 @@ Expect to NOT see `+1 412-612-9992` in your server logs on your callback.
The `pii_masking` guardrail ran on this request because api key=sk-jNm1Zar7XfNdZXp49Z1kSQ has `"permissions": {"pii_masking": true}`
:::
## Specification
## Specification
### `guardrails` Configuration on YAML
@ -723,6 +736,7 @@ The `guardrails` parameter can be passed to any LiteLLM Proxy endpoint (`/chat/c
#### Format Options
1. Simple List Format:
```python
"guardrails": [
"aporia-pre-guard",
@ -730,9 +744,10 @@ The `guardrails` parameter can be passed to any LiteLLM Proxy endpoint (`/chat/c
]
```
2. Advanced Dictionary Format:
1. Advanced Dictionary Format:
In this format the dictionary key is `guardrail_name` you want to run
```python
"guardrails": {
"aporia-pre-guard": {
@ -745,6 +760,7 @@ In this format the dictionary key is `guardrail_name` you want to run
```
#### Type Definition
```python
guardrails: Union[
List[str], # Simple list of guardrail names
@ -754,3 +770,4 @@ guardrails: Union[
class DynamicGuardrailParams:
extra_body: Dict[str, Any] # Additional parameters for the guardrail
```

BIN
docs/my-website/img/april_townhall_isolated_environments.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 312 KiB

BIN
docs/my-website/img/skip_system_message_guardrail_ui.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 126 KiB

BIN
docs/my-website/img/stable_main.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 236 KiB

BIN
docs/my-website/img/verify_releases.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 1.3 MiB

20
docs/my-website/sidebars.js
View file

@ -350,6 +350,7 @@ const sidebars = {
"proxy/debugging",
"proxy/error_diagnosis",
"proxy/deploy",
"proxy/docker_image_security",
"proxy/health",
"proxy/master_key_rotations",
"proxy/model_management",
@ -564,7 +565,8 @@ const sidebars = {
"proxy/model_access",
"proxy/model_access_groups",
"proxy/access_groups",
"proxy/team_model_add"
"proxy/team_model_add",
"proxy/credential_routing"
]
},
{
@ -860,6 +862,7 @@ const sidebars = {
]
},
"providers/anthropic",
"providers/anthropic_tool_search",
"providers/aws_sagemaker",
{
type: "category",
@ -1057,16 +1060,7 @@ const sidebars = {
"proxy/health_check_routing"
],
},
{
type: "category",
label: "Load Testing",
items: [
"benchmarks",
"load_test_advanced",
"load_test_sdk",
"load_test_rpm",
]
},
"benchmarks",
{
type: "category",
label: "Contributing",
@ -1095,6 +1089,9 @@ const sidebars = {
"data_retention",
"proxy/security_encryption_faq",
"migration_policy",
"load_test_advanced",
"load_test_sdk",
"load_test_rpm",
{
type: "category",
label: "❤️ 🚅 Projects built on LiteLLM",
@ -1231,6 +1228,7 @@ const learnSidebar = {
"completion/web_fetch",
"completion/computer_use",
"guides/code_interpreter",
"completion/anthropic_advisor_tool",
"completion/message_sanitization",
],
},

120
docs/my-website/src/css/custom.css
View file

@ -794,3 +794,123 @@ video {
max-width: calc(9 / 12 * 100%) !important;
}
}
/* =========================================
BLOG — Ramp-style aesthetic
========================================= */
/* Hide blog sidebar on post pages */
.blog-post-page aside.col {
display: none !important;
}
/* Make blog post content full-width + constrained */
.blog-post-page main.col--7 {
--ifm-col-width: 100% !important;
max-width: 820px !important;
margin: 0 auto !important;
flex: 0 0 100% !important;
}
/* Clean post header */
.blog-wrapper article header h1 {
font-size: 2rem;
font-weight: 600;
letter-spacing: -0.02em;
line-height: 1.25;
color: #111827;
margin-bottom: 0.75rem;
}
/* Author / date line */
.blog-wrapper article header .avatar,
.blog-wrapper article header [class*='blogPostData'] {
margin-top: 0.75rem;
}
/* Clean prose body */
.blog-wrapper article .markdown {
font-size: 0.95rem;
line-height: 1.7;
color: #374151;
}
.blog-wrapper article .markdown h2 {
font-size: 1.35rem;
font-weight: 600;
letter-spacing: -0.01em;
margin-top: 2.5rem;
margin-bottom: 0.75rem;
color: #111827;
}
.blog-wrapper article .markdown h3 {
font-size: 1.1rem;
font-weight: 600;
margin-top: 2rem;
margin-bottom: 0.5rem;
color: #111827;
}
.blog-wrapper article .markdown p {
margin-bottom: 1.25rem;
}
.blog-wrapper article .markdown a {
color: #0ea5e9;
text-decoration: none;
}
.blog-wrapper article .markdown a:hover {
text-decoration: underline;
}
.blog-wrapper article .markdown code {
font-size: 0.85em;
background: #f3f4f6;
border: 1px solid #e5e7eb;
border-radius: 4px;
padding: 0.15em 0.4em;
color: #111827;
}
.blog-wrapper article .markdown pre {
background: #ffffff !important;
border: 1px solid #e5e7eb !important;
border-radius: 8px;
box-shadow: 0 1px 3px rgba(0,0,0,0.06);
}
.blog-wrapper article .markdown pre code {
background: transparent;
border: none;
padding: 0;
color: inherit;
}
/* Hide tags section at bottom of blog posts */
.blog-wrapper footer [class*='blogPostTags'],
.blog-wrapper footer [class*='tags'] {
display: none;
}
/* Nav buttons (prev/next) at bottom - keep clean */
.blog-wrapper .pagination-nav__label {
font-size: 0.85rem;
}
[data-theme='dark'] .blog-wrapper article header h1,
[data-theme='dark'] .blog-wrapper article .markdown h2,
[data-theme='dark'] .blog-wrapper article .markdown h3 {
color: #f9fafb;
}
[data-theme='dark'] .blog-wrapper article .markdown {
color: #d1d5db;
}
[data-theme='dark'] .blog-wrapper article .markdown code {
background: #1f2937;
border-color: #374151;
color: #f9fafb;
}

192
docs/my-website/src/theme/BlogListPage/index.js
View file

@ -3,78 +3,86 @@ import Layout from '@theme/Layout';
import Link from '@docusaurus/Link';
import styles from './styles.module.css';
const TAG_COLORS = {
gemini: {bg: '#d2e3fc', text: '#174ea6', darkBg: '#1a3a5c', darkText: '#8ab4f8'},
anthropic: {bg: '#fde0c4', text: '#b33d00', darkBg: '#4a2800', darkText: '#ffb74d'},
claude: {bg: '#fde0c4', text: '#b33d00', darkBg: '#4a2800', darkText: '#ffb74d'},
llms: {bg: '#c8e6c9', text: '#1b5e20', darkBg: '#1b3d1f', darkText: '#81c784'},
};
// ── Provider marquee ──────────────────────────────────────────────────────
const PROVIDERS = [
{ name: 'OpenAI', img: 'https://www.google.com/s2/favicons?domain=openai.com&sz=64' },
{ name: 'Anthropic', img: 'https://www.google.com/s2/favicons?domain=claude.ai&sz=64' },
{ name: 'Google Gemini', img: 'https://www.google.com/s2/favicons?domain=ai.google.dev&sz=64' },
{ name: 'AWS Bedrock', img: 'https://www.google.com/s2/favicons?domain=aws.amazon.com&sz=64' },
{ name: 'Azure OpenAI', img: 'https://www.google.com/s2/favicons?domain=azure.microsoft.com&sz=64' },
{ name: 'Mistral AI', img: 'https://www.google.com/s2/favicons?domain=mistral.ai&sz=64' },
{ name: 'Meta Llama', img: 'https://www.google.com/s2/favicons?domain=meta.com&sz=64' },
{ name: 'Groq', img: 'https://www.google.com/s2/favicons?domain=groq.com&sz=64' },
{ name: 'Hugging Face', img: 'https://www.google.com/s2/favicons?domain=huggingface.co&sz=64' },
{ name: 'Perplexity', img: 'https://www.google.com/s2/favicons?domain=perplexity.ai&sz=64' },
{ name: 'DeepSeek', img: 'https://www.google.com/s2/favicons?domain=deepseek.com&sz=64' },
{ name: 'Cohere', img: 'https://www.google.com/s2/favicons?domain=cohere.com&sz=64' },
{ name: 'Together AI', img: 'https://www.google.com/s2/favicons?domain=together.ai&sz=64' },
{ name: 'Vertex AI', img: 'https://www.google.com/s2/favicons?domain=cloud.google.com&sz=64' },
];
function hashHue(str) {
let hash = 0;
for (let i = 0; i < str.length; i++) {
hash = str.charCodeAt(i) + ((hash << 5) - hash);
}
return Math.abs(hash) % 360;
}
function getTagColor(label) {
const key = label.toLowerCase();
for (const [k, v] of Object.entries(TAG_COLORS)) {
if (key === k) return v;
}
const hue = hashHue(key);
return {
bg: `hsl(${hue}, 40%, 90%)`,
text: `hsl(${hue}, 60%, 25%)`,
darkBg: `hsl(${hue}, 40%, 20%)`,
darkText: `hsl(${hue}, 50%, 75%)`,
};
}
function formatDate(dateStr) {
const d = new Date(dateStr);
const now = new Date();
const diffDays = Math.floor((now - d) / (1000 * 60 * 60 * 24));
if (diffDays <= 0) return 'Today';
if (diffDays === 1) return '1d ago';
if (diffDays < 30) return `${diffDays}d ago`;
return d.toLocaleDateString('en-US', {month: 'short', day: 'numeric', year: 'numeric'});
}
function BlogCard({post, featured}) {
const {title, permalink, date, description, tags} = post;
const visibleTags = (tags || []).slice(0, 3);
const DOUBLED = [...PROVIDERS, ...PROVIDERS];
function ProviderMarquee() {
return (
<Link to={permalink} className={styles.cardLink} aria-label={title}>
<article className={featured ? styles.cardFeatured : styles.card}>
<div className={styles.meta}>
<time className={styles.time} dateTime={date}>{formatDate(date)}</time>
{featured && <span className={styles.badge}>Latest</span>}
<div className={styles.marqueeWrap}>
<p className={styles.marqueeLabel}>Routing to 100+ providers</p>
<div className={styles.marqueeOuter}>
<div className={styles.fadeLeft} />
<div className={styles.fadeRight} />
<div className={styles.marqueeTrack}>
{DOUBLED.map((p, i) => (
<span key={i} className={styles.marqueeItem}>
<img src={p.img} alt={p.name} width={18} height={18} className={styles.marqueeIcon} />
<span>{p.name}</span>
<span className={styles.marqueeSep}>|</span>
</span>
))}
</div>
</div>
</div>
);
}
// ── Post row ──────────────────────────────────────────────────────────────
function formatDate(dateStr) {
return new Date(dateStr).toLocaleDateString('en-US', {
month: 'long', day: 'numeric', year: 'numeric',
});
}
function AuthorList({authors}) {
if (!authors || authors.length === 0) return null;
return (
<>
{authors.map((a, i) => (
<React.Fragment key={a.name}>
{i > 0 && <span className={styles.authorSep}> </span>}
{a.url ? (
<a href={a.url} target="_blank" rel="noopener" className={styles.authorLink}>{a.name}</a>
) : (
<span className={styles.authorName}>{a.name}</span>
)}
</React.Fragment>
))}
</>
);
}
function PostRow({post}) {
const {title, permalink, date, description, authors} = post;
return (
<article className={styles.post}>
<Link to={permalink} className={styles.titleLink}>
<h2 className={styles.title}>{title}</h2>
{description && <p className={styles.desc}>{description}</p>}
{visibleTags.length > 0 && (
<div className={styles.tags}>
{visibleTags.map(tag => {
const c = getTagColor(tag.label);
return (
<span key={tag.label} className={styles.tag} style={{
'--tag-bg': c.bg, '--tag-text': c.text,
'--tag-bg-dark': c.darkBg, '--tag-text-dark': c.darkText,
}}>{tag.label}</span>
);
})}
</div>
)}
<div className={styles.arrow} aria-hidden="true">
<svg width="16" height="16" viewBox="0 0 16 16" fill="none">
<path d="M6 3l5 5-5 5" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" strokeLinejoin="round"/>
</svg>
</div>
</article>
</Link>
</Link>
{description && <p className={styles.desc}>{description}</p>}
<div className={styles.meta}>
<AuthorList authors={authors} />
{authors && authors.length > 0 && <span className={styles.metaDash}> — </span>}
<time className={styles.date} dateTime={date}>{formatDate(date)}</time>
</div>
</article>
);
}
@ -83,41 +91,47 @@ function Pagination({metadata}) {
if (!previousPage && !nextPage) return null;
return (
<nav className={styles.pagination} aria-label="Blog list pagination">
{previousPage ? (
<Link to={previousPage} className={styles.paginationLink}>&larr; Newer posts</Link>
) : <span />}
{nextPage ? (
<Link to={nextPage} className={styles.paginationLink}>Older posts &rarr;</Link>
) : <span />}
{previousPage ? <Link to={previousPage} className={styles.pageLink}>&larr; Newer posts</Link> : <span />}
{nextPage ? <Link to={nextPage} className={styles.pageLink}>Older posts &rarr;</Link> : <span />}
</nav>
);
}
// ── Page ──────────────────────────────────────────────────────────────────
export default function BlogListPage(props) {
const items = props.items || [];
const metadata = props.metadata || {};
const [first, ...rest] = items;
return (
<Layout
title={metadata.blogTitle || 'Blog'}
description={metadata.blogDescription || 'Guides, announcements, and best practices from the LiteLLM team.'}
title="Engineering Blog"
description="How we build the world's most widely used open-source AI Gateway. Routing, reliability, observability, and what we learn along the way."
>
<header className={styles.hero}>
<h1 className={styles.heroTitle}>The LiteLLM Blog</h1>
<p className={styles.heroSubtitle}>Guides, announcements, and best practices from the LiteLLM team.</p>
</header>
<div className={styles.page}>
{/* Hero */}
<header className={styles.hero}>
<p className={styles.eyebrow}>AI Gateway</p>
<h1 className={styles.heroTitle}>Engineering</h1>
<p className={styles.heroSub}>
How we build the world's most widely used open-source AI Gateway.
Routing, reliability, observability, and what we learn along the way.
</p>
<a href="https://jobs.ashbyhq.com/litellm" target="_blank" rel="noopener noreferrer" className={styles.hiringBtn}>
We're hiring!
</a>
</header>
<main className={styles.grid}>
{first && (
<BlogCard post={first.content.metadata} featured />
)}
{rest.map(({content}) => (
<BlogCard key={content.metadata.permalink} post={content.metadata} />
))}
</main>
<ProviderMarquee />
<Pagination metadata={metadata} />
{/* Post list */}
<main className={styles.list}>
{items.map(({content}) => (
<PostRow key={content.metadata.permalink} post={content.metadata} />
))}
</main>
<Pagination metadata={metadata} />
</div>
</Layout>
);
}

353
docs/my-website/src/theme/BlogListPage/styles.module.css
View file

@ -1,163 +1,254 @@
.hero {
max-width: 960px;
/* ── Page shell ───────────────────────────────────────────────────────── */
.page {
max-width: 860px;
margin: 0 auto;
padding: 3rem 1.5rem 1rem;
text-align: center;
padding: 0 2rem;
}
/* ── Hero ─────────────────────────────────────────────────────────────── */
.hero {
padding: 3.5rem 0 0;
}
.eyebrow {
font-size: 0.68rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.12em;
color: #0ea5e9;
margin: 0 0 0.5rem;
}
.heroTitle {
font-size: 2.25rem;
font-weight: 700;
margin-bottom: 0.25rem;
letter-spacing: -0.02em;
}
.heroSubtitle {
color: var(--ifm-color-emphasis-600);
font-size: 1.1rem;
margin-bottom: 0;
}
.grid {
max-width: 960px;
margin: 0 auto;
padding: 1.5rem;
display: grid;
gap: 1rem;
}
.cardLink {
display: block;
text-decoration: none;
color: inherit;
}
.card {
position: relative;
border: 1px solid var(--ifm-color-emphasis-200);
border-radius: 12px;
padding: 1.5rem;
padding-right: 2.5rem;
height: 100%;
transition: border-color 0.15s, transform 0.15s, background 0.15s;
background: var(--ifm-background-surface-color, var(--ifm-background-color));
}
.card:hover {
border-color: var(--ifm-color-primary);
transform: translateY(-2px);
background: var(--ifm-color-emphasis-100);
}
.cardFeatured {
composes: card;
border-color: var(--ifm-color-primary-lighter);
background: var(--ifm-color-emphasis-100);
}
.meta {
display: flex;
align-items: center;
gap: 0.5rem;
margin-bottom: 0.5rem;
}
.time {
font-size: 0.8rem;
font-weight: 500;
color: var(--ifm-color-emphasis-600);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.badge {
font-size: 0.65rem;
font-size: 2.75rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.06em;
padding: 2px 8px;
border-radius: 99px;
background: var(--ifm-color-primary);
color: #fff;
}
.title {
font-size: 1.15rem;
font-weight: 600;
margin: 0 0 0.4rem;
line-height: 1.35;
}
.desc {
font-size: 0.88rem;
color: var(--ifm-color-emphasis-700);
line-height: 1.5;
letter-spacing: -0.03em;
line-height: 1.1;
color: #111827;
margin: 0 0 0.75rem;
}
.tags {
display: flex;
gap: 6px;
flex-wrap: wrap;
.heroSub {
font-size: 0.95rem;
color: #6b7280;
max-width: 540px;
line-height: 1.65;
margin: 0 0 1.25rem;
}
.tag {
font-size: 0.7rem;
.hiringBtn {
display: inline-block;
background: #111827;
color: #fff !important;
font-size: 0.82rem;
font-weight: 500;
padding: 2px 10px;
border-radius: 99px;
background: var(--tag-bg);
color: var(--tag-text);
padding: 0.45rem 1rem;
border-radius: 6px;
text-decoration: none !important;
transition: background 0.15s;
}
:global([data-theme='dark']) .tag {
background: var(--tag-bg-dark);
color: var(--tag-text-dark);
.hiringBtn:hover {
background: #000;
}
.arrow {
/* ── Marquee ──────────────────────────────────────────────────────────── */
.marqueeWrap {
margin: 2.5rem 0 0;
padding: 1.25rem 0;
border-top: 1px solid #f3f4f6;
border-bottom: 1px solid #f3f4f6;
overflow: hidden;
}
.marqueeLabel {
text-align: center;
font-size: 0.62rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.14em;
color: #9ca3af;
margin: 0 0 1rem;
}
.marqueeOuter {
position: relative;
overflow: hidden;
}
.fadeLeft {
pointer-events: none;
position: absolute;
right: 1rem;
top: 50%;
transform: translateY(-50%);
color: var(--ifm-color-emphasis-400);
transition: color 0.15s, transform 0.15s;
left: 0; top: 0; bottom: 0;
width: 5rem;
background: linear-gradient(to right, var(--ifm-background-color, #fff), transparent);
z-index: 10;
}
.card:hover .arrow {
color: var(--ifm-color-primary);
transform: translateY(-50%) translateX(3px);
.fadeRight {
pointer-events: none;
position: absolute;
right: 0; top: 0; bottom: 0;
width: 5rem;
background: linear-gradient(to left, var(--ifm-background-color, #fff), transparent);
z-index: 10;
}
.marqueeTrack {
display: flex;
align-items: center;
white-space: nowrap;
animation: marquee 28s linear infinite;
}
@keyframes marquee {
from { transform: translateX(0); }
to { transform: translateX(-50%); }
}
.marqueeItem {
display: inline-flex;
align-items: center;
gap: 0.45rem;
padding: 0 1.4rem;
font-size: 0.82rem;
color: #4b5563;
font-weight: 500;
}
.marqueeIcon {
flex-shrink: 0;
border-radius: 2px;
}
.marqueeSep {
margin-left: 1.2rem;
color: #e5e7eb;
font-weight: 300;
}
/* ── Post list ────────────────────────────────────────────────────────── */
.list {
margin-top: 0.5rem;
}
.post {
padding: 2.25rem 0;
border-bottom: 1px solid #f3f4f6;
}
.titleLink {
text-decoration: none !important;
color: inherit;
}
.title {
font-size: 1.4rem;
font-weight: 600;
line-height: 1.3;
letter-spacing: -0.01em;
color: #111827;
margin: 0 0 0.5rem;
transition: color 0.12s;
}
.titleLink:hover .title {
color: #0ea5e9;
}
.desc {
font-size: 0.875rem;
color: #6b7280;
line-height: 1.55;
margin: 0 0 0.6rem;
}
.meta {
font-size: 0.82rem;
color: #6b7280;
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0;
}
.authorLink {
color: #374151;
font-weight: 500;
text-decoration: underline;
text-underline-offset: 2px;
text-decoration-color: #d1d5db;
}
.authorLink:hover {
color: #0ea5e9;
text-decoration-color: #0ea5e9;
}
.authorName {
color: #374151;
font-weight: 500;
}
.authorSep {
margin: 0 0.3rem;
color: #d1d5db;
}
.metaDash {
margin: 0 0.35rem;
color: #d1d5db;
}
.date {
color: #9ca3af;
}
/* ── Pagination ───────────────────────────────────────────────────────── */
.pagination {
max-width: 960px;
margin: 0 auto;
padding: 1rem 1.5rem 3rem;
padding: 1.5rem 0 4rem;
display: flex;
justify-content: space-between;
}
.paginationLink {
font-size: 0.9rem;
.pageLink {
font-size: 0.85rem;
font-weight: 500;
color: var(--ifm-color-primary);
color: #374151;
text-decoration: none;
}
.paginationLink:hover {
text-decoration: underline;
.pageLink:hover {
color: #0ea5e9;
}
@media (min-width: 640px) {
.grid {
grid-template-columns: repeat(2, 1fr);
}
.grid .cardLink:first-child {
grid-column: 1 / -1;
}
.grid .cardLink:last-child:nth-child(even) {
grid-column: 1 / -1;
}
/* ── Dark mode ────────────────────────────────────────────────────────── */
[data-theme='dark'] .heroTitle,
[data-theme='dark'] .title {
color: #f9fafb;
}
[data-theme='dark'] .heroSub,
[data-theme='dark'] .desc,
[data-theme='dark'] .date {
color: #9ca3af;
}
[data-theme='dark'] .post,
[data-theme='dark'] .marqueeWrap {
border-color: #1f2937;
}
[data-theme='dark'] .authorLink,
[data-theme='dark'] .authorName {
color: #e5e7eb;
}
[data-theme='dark'] .hiringBtn {
background: #f9fafb;
color: #111827 !important;
}
[data-theme='dark'] .hiringBtn:hover {
background: #fff;
}

54
docs/my-website/src/theme/BlogPostPage/index.js Normal file
View file

@ -0,0 +1,54 @@
import React, {useEffect} from 'react';
import OriginalBlogPostPage from '@theme-original/BlogPostPage';
import styles from './styles.module.css';
function BackLink() {
return (
<div className={styles.backOuter}>
<a href="/blog" className={styles.backLink}>
<svg className={styles.backArrow} fill="none" stroke="currentColor" viewBox="0 0 24 24" aria-hidden="true">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M7 16l-4-4m0 0l4-4m-4 4h18" />
</svg>
Blog
</a>
</div>
);
}
function HiringCTA() {
return (
<div className={styles.ctaOuter}>
<div className={styles.cta}>
<p className={styles.ctaEyebrow}>We're hiring</p>
<a
href="https://jobs.ashbyhq.com/litellm"
target="_blank"
rel="noopener noreferrer"
className={styles.ctaLink}
>
Like what you see? Join us
<svg className={styles.ctaArrow} fill="none" stroke="currentColor" viewBox="0 0 24 24" aria-hidden="true">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M17 8l4 4m0 0l-4 4m4-4H3" />
</svg>
</a>
<p className={styles.ctaSub}>Come build the future of AI infrastructure.</p>
</div>
</div>
);
}
export default function BlogPostPage(props) {
// Add body class so CSS can hide the sidebar
useEffect(() => {
document.body.classList.add('blog-post-body');
return () => document.body.classList.remove('blog-post-body');
}, []);
return (
<>
<BackLink />
<OriginalBlogPostPage {...props} />
<HiringCTA />
</>
);
}

109
docs/my-website/src/theme/BlogPostPage/styles.module.css Normal file
View file

@ -0,0 +1,109 @@
.backOuter {
position: fixed;
top: calc(var(--ifm-navbar-height, 60px) + 1rem);
left: 2rem;
z-index: 100;
}
.backLink {
display: inline-flex;
align-items: center;
gap: 0.375rem;
font-size: 0.875rem;
font-weight: 500;
color: #6b7280;
text-decoration: none !important;
transition: color 0.15s;
}
.backLink:hover {
color: #111827;
}
.backArrow {
width: 1rem;
height: 1rem;
transition: transform 0.15s;
flex-shrink: 0;
}
.backLink:hover .backArrow {
transform: translateX(-3px);
}
[data-theme='dark'] .backLink {
color: #9ca3af;
}
[data-theme='dark'] .backLink:hover {
color: #f9fafb;
}
.ctaOuter {
max-width: 820px;
margin: 0 auto;
padding: 0 2rem 4rem;
}
.cta {
border-radius: 16px;
background: #f9fafb;
border: 1px solid #e5e7eb;
padding: 2.5rem 2rem;
text-align: center;
}
.ctaEyebrow {
font-size: 0.68rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.12em;
color: #9ca3af;
margin: 0 0 0.75rem;
}
.ctaLink {
display: inline-flex;
align-items: center;
gap: 0.5rem;
font-size: 1.5rem;
font-weight: 600;
letter-spacing: -0.01em;
color: #111827;
text-decoration: none !important;
transition: color 0.15s;
}
.ctaLink:hover {
color: #0ea5e9;
}
.ctaArrow {
width: 1.25rem;
height: 1.25rem;
transition: transform 0.15s;
flex-shrink: 0;
}
.ctaLink:hover .ctaArrow {
transform: translateX(3px);
}
.ctaSub {
margin: 0.75rem 0 0;
font-size: 0.875rem;
color: #6b7280;
}
[data-theme='dark'] .cta {
background: #1f2937;
border-color: #374151;
}
[data-theme='dark'] .ctaLink {
color: #f9fafb;
}
[data-theme='dark'] .ctaSub {
color: #9ca3af;
}

2
enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py
View file

@ -114,6 +114,7 @@ class PagerDutyAlerting(SlackAlerting):
user_api_key_max_budget=_meta.get("user_api_key_max_budget"),
user_api_key_budget_reset_at=_meta.get("user_api_key_budget_reset_at"),
user_api_key_org_id=_meta.get("user_api_key_org_id"),
user_api_key_org_alias=_meta.get("user_api_key_org_alias"),
user_api_key_team_id=_meta.get("user_api_key_team_id"),
user_api_key_project_id=_meta.get("user_api_key_project_id"),
user_api_key_project_alias=_meta.get("user_api_key_project_alias"),
@ -196,6 +197,7 @@ class PagerDutyAlerting(SlackAlerting):
else None
),
user_api_key_org_id=user_api_key_dict.org_id,
user_api_key_org_alias=user_api_key_dict.organization_alias,
user_api_key_team_id=user_api_key_dict.team_id,
user_api_key_project_id=user_api_key_dict.project_id,
user_api_key_project_alias=user_api_key_dict.project_alias,

3
litellm/__init__.py
View file

@ -164,6 +164,7 @@ initialized_langfuse_clients: int = 0
langfuse_default_tags: Optional[List[str]] = None
langsmith_batch_size: Optional[int] = None
prometheus_initialize_budget_metrics: Optional[bool] = False
prometheus_latency_buckets: Optional[List[float]] = None
require_auth_for_metrics_endpoint: Optional[bool] = False
argilla_batch_size: Optional[int] = None
datadog_use_v1: Optional[bool] = False # if you want to use v1 datadog logged payload.
@ -203,6 +204,7 @@ add_user_information_to_llm_headers: Optional[
bool
] = None # adds user_id, team_id, token hash (params from StandardLoggingMetadata) to request headers
store_audit_logs = False # Enterprise feature, allow users to see audit logs
skip_system_message_in_guardrail: bool = False
### end of callbacks #############
email: Optional[
@ -318,6 +320,7 @@ return_response_headers: bool = (
False # get response headers from LLM Api providers - example x-remaining-requests,
)
enable_json_schema_validation: bool = False
enable_model_config_credential_overrides: bool = False
enable_key_alias_format_validation: bool = (
False # opt-in validation of key_alias format on /key/generate and /key/update
)

6
litellm/_logging.py
View file

@ -243,6 +243,12 @@ class JsonFormatter(Formatter):
if key not in _STANDARD_RECORD_ATTRS and key not in json_record:
json_record[key] = value
# Set component/logger only if not already supplied via extra={...}
if "component" not in json_record:
json_record["component"] = record.name
if "logger" not in json_record:
json_record["logger"] = f"{record.filename}:{record.lineno}"
if record.exc_info:
json_record["stacktrace"] = record.exc_text or self.formatException(
record.exc_info

6
litellm/a2a_protocol/main.py
View file

@ -615,7 +615,7 @@ async def asend_message_streaming( # noqa: PLR0915
async def create_a2a_client(
base_url: str,
timeout: float = 60.0,
timeout: float = DEFAULT_A2A_AGENT_TIMEOUT,
extra_headers: Optional[Dict[str, str]] = None,
) -> "A2AClientType":
"""
@ -626,7 +626,7 @@ async def create_a2a_client(
Args:
base_url: The base URL of the A2A agent (e.g., "http://localhost:10001")
timeout: Request timeout in seconds (default: 60.0)
timeout: Request timeout in seconds (default: ``DEFAULT_A2A_AGENT_TIMEOUT`` / env ``DEFAULT_A2A_AGENT_TIMEOUT``)
extra_headers: Optional additional headers to include in requests
Returns:
@ -711,7 +711,7 @@ async def aget_agent_card(
Args:
base_url: The base URL of the A2A agent (e.g., "http://localhost:10001")
timeout: Request timeout in seconds (default: 60.0)
timeout: Request timeout in seconds (default: ``DEFAULT_A2A_AGENT_TIMEOUT`` / env ``DEFAULT_A2A_AGENT_TIMEOUT``)
extra_headers: Optional additional headers to include in requests
Returns:

6
litellm/a2a_protocol/providers/pydantic_ai_agents/config.py
View file

@ -21,11 +21,11 @@ class PydanticAIProviderConfig(BaseA2AProviderConfig):
request_id: str,
params: Dict[str, Any],
api_base: Optional[str] = None,
**kwargs,
**kwargs: Any,
) -> Dict[str, Any]:
"""Handle non-streaming request to Pydantic AI agent."""
if not api_base:
raise ValueError("api_base is required for Pydantic AI agents")
if api_base is None:
raise ValueError("api_base is required for PydanticAIProviderConfig")
return await PydanticAIHandler.handle_non_streaming(
request_id=request_id,
params=params,

6
litellm/anthropic_beta_headers_config.json
View file

@ -1,6 +1,7 @@
{
"description": "Mapping of Anthropic beta headers for each provider. Keys are input header names, values are provider-specific header names (or null if unsupported). Only headers present in mapping keys with non-null values can be forwarded.",
"anthropic": {
"advisor-tool-2026-03-01": "advisor-tool-2026-03-01",
"advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
"bash_20241022": null,
"bash_20250124": null,
@ -31,6 +32,7 @@
"web-search-2025-03-05": "web-search-2025-03-05"
},
"azure_ai": {
"advisor-tool-2026-03-01": null,
"advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
"bash_20241022": null,
"bash_20250124": null,
@ -60,6 +62,7 @@
"web-search-2025-03-05": "web-search-2025-03-05"
},
"bedrock_converse": {
"advisor-tool-2026-03-01": null,
"advanced-tool-use-2025-11-20": null,
"bash_20241022": null,
"bash_20250124": null,
@ -90,6 +93,7 @@
"web-search-2025-03-05": null
},
"bedrock": {
"advisor-tool-2026-03-01": null,
"advanced-tool-use-2025-11-20": "tool-search-tool-2025-10-19",
"bash_20241022": null,
"bash_20250124": null,
@ -120,6 +124,7 @@
"web-search-2025-03-05": null
},
"vertex_ai": {
"advisor-tool-2026-03-01": null,
"advanced-tool-use-2025-11-20": "tool-search-tool-2025-10-19",
"bash_20241022": null,
"bash_20250124": null,
@ -150,6 +155,7 @@
"web-search-2025-03-05": "web-search-2025-03-05"
},
"databricks": {
"advisor-tool-2026-03-01": null,
"advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
"bash_20241022": null,
"bash_20250124": null,

9
litellm/constants.py
View file

@ -135,6 +135,15 @@ MCP_OAUTH2_TOKEN_CACHE_DEFAULT_TTL = int(
MCP_NPM_CACHE_DIR = os.getenv("MCP_NPM_CACHE_DIR", "/tmp/.npm_mcp_cache")
MCP_OAUTH2_TOKEN_CACHE_MIN_TTL = int(os.getenv("MCP_OAUTH2_TOKEN_CACHE_MIN_TTL", "10"))
# Per-user OAuth token Redis cache (for server-side token storage)
MCP_PER_USER_TOKEN_REDIS_KEY_PREFIX = "mcp:per_user_token"
MCP_PER_USER_TOKEN_DEFAULT_TTL = int(
os.getenv("MCP_PER_USER_TOKEN_DEFAULT_TTL", "43200") # 12 hours
)
MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS = int(
os.getenv("MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS", "60")
)
# MCP timeout defaults (seconds). Override via env vars for slow/custom MCP servers.
MCP_CLIENT_TIMEOUT = float(os.getenv("LITELLM_MCP_CLIENT_TIMEOUT", "60.0"))
MCP_TOOL_LISTING_TIMEOUT = float(os.getenv("LITELLM_MCP_TOOL_LISTING_TIMEOUT", "30.0"))

26
litellm/containers/endpoint_factory.py
View file

@ -14,6 +14,7 @@ from typing import Any, Callable, Dict, List, Literal, Optional, Type
import litellm
from litellm.constants import request_timeout as DEFAULT_REQUEST_TIMEOUT
from litellm.containers.utils import decode_managed_container_id_for_request
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.containers.transformation import BaseContainerConfig
from litellm.llms.custom_httpx.container_handler import generic_container_handler
@ -53,7 +54,7 @@ def create_sync_endpoint_function(endpoint_config: Dict) -> Callable:
@client
def endpoint_func(
timeout: int = 600,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: Optional[Dict[str, Any]] = None,
extra_query: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,
@ -61,6 +62,7 @@ def create_sync_endpoint_function(endpoint_config: Dict) -> Callable:
):
local_vars = locals()
try:
resolved_custom_llm_provider: str = custom_llm_provider
litellm_logging_obj: LiteLLMLoggingObj = kwargs.pop("litellm_logging_obj")
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id")
_is_async = kwargs.pop("async_call", False) is True
@ -76,15 +78,27 @@ def create_sync_endpoint_function(endpoint_config: Dict) -> Callable:
# Get provider config
litellm_params = GenericLiteLLMParams(**kwargs)
# Strip LiteLLM-managed container IDs before calling the provider API
# (OpenAI enforces max length 64 on container_id).
if "container_id" in kwargs and isinstance(kwargs["container_id"], str):
(
kwargs["container_id"],
resolved_custom_llm_provider,
litellm_params,
) = decode_managed_container_id_for_request(
container_id=kwargs["container_id"],
custom_llm_provider=resolved_custom_llm_provider,
litellm_params=litellm_params,
)
container_provider_config: Optional[
BaseContainerConfig
] = ProviderConfigManager.get_provider_container_config(
provider=litellm.LlmProviders(custom_llm_provider),
provider=litellm.LlmProviders(resolved_custom_llm_provider),
)
if container_provider_config is None:
raise ValueError(
f"Container provider config not found for: {custom_llm_provider}"
f"Container provider config not found for: {resolved_custom_llm_provider}"
)
# Build optional params for logging
@ -96,7 +110,7 @@ def create_sync_endpoint_function(endpoint_config: Dict) -> Callable:
model="",
optional_params=optional_params,
litellm_params={"litellm_call_id": litellm_call_id},
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
)
# Use generic handler
@ -115,7 +129,7 @@ def create_sync_endpoint_function(endpoint_config: Dict) -> Callable:
except Exception as e:
raise litellm.exception_type(
model="",
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
original_exception=e,
completion_kwargs=local_vars,
extra_kwargs=kwargs,
@ -133,7 +147,7 @@ def create_async_endpoint_function(
@client
async def async_endpoint_func(
timeout: int = 600,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: Optional[Dict[str, Any]] = None,
extra_query: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,

202
litellm/containers/main.py
View file

@ -6,7 +6,10 @@ from typing import Any, Coroutine, Dict, List, Literal, Optional, Union, overloa
import litellm
from litellm.constants import request_timeout as DEFAULT_REQUEST_TIMEOUT
from litellm.containers.utils import ContainerRequestUtils
from litellm.containers.utils import (
ContainerRequestUtils,
decode_managed_container_id_for_request,
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.containers.transformation import BaseContainerConfig
from litellm.main import base_llm_http_handler
@ -48,7 +51,7 @@ async def acreate_container(
file_ids: Optional[List[str]] = None,
timeout=600, # default to 10 minutes
# LiteLLM specific params,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -122,7 +125,7 @@ def create_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
acreate_container: Literal[True],
**kwargs,
@ -139,7 +142,7 @@ def create_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
acreate_container: Literal[False] = False,
**kwargs,
@ -158,7 +161,7 @@ def create_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -247,7 +250,7 @@ def create_container(
# Set the correct call type for container creation
litellm_logging_obj.call_type = CallTypes.create_container.value
return base_llm_http_handler.container_create_handler(
container_obj = base_llm_http_handler.container_create_handler(
name=name,
container_create_request_params=container_create_request_params,
container_provider_config=container_provider_config,
@ -257,6 +260,17 @@ def create_container(
timeout=timeout or DEFAULT_REQUEST_TIMEOUT,
_is_async=_is_async,
)
# Encode container_id with provider/model metadata for routing
if isinstance(container_obj, ContainerObject):
container_obj = ContainerRequestUtils.encode_container_id_in_response(
response_obj=container_obj,
custom_llm_provider=custom_llm_provider,
litellm_metadata=kwargs.get("litellm_metadata"),
extra_body=extra_body,
)
return container_obj
except Exception as e:
raise litellm.exception_type(
@ -275,7 +289,7 @@ async def alist_containers(
limit: Optional[int] = None,
order: Optional[str] = None,
timeout=600, # default to 10 minutes
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -348,7 +362,7 @@ def list_containers(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
alist_containers: Literal[True],
**kwargs,
@ -365,7 +379,7 @@ def list_containers(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
alist_containers: Literal[False] = False,
**kwargs,
@ -384,7 +398,7 @@ def list_containers(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -481,7 +495,7 @@ def list_containers(
async def aretrieve_container(
container_id: str,
timeout=600, # default to 10 minutes
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -548,7 +562,7 @@ def retrieve_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
aretrieve_container: Literal[True],
**kwargs,
@ -563,7 +577,7 @@ def retrieve_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
aretrieve_container: Literal[False] = False,
**kwargs,
@ -580,7 +594,7 @@ def retrieve_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -594,6 +608,7 @@ def retrieve_container(
"""
local_vars = locals()
try:
resolved_custom_llm_provider: str = custom_llm_provider
litellm_logging_obj: LiteLLMLoggingObj = kwargs.pop("litellm_logging_obj") # type: ignore
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id")
_is_async = kwargs.pop("async_call", False) is True
@ -615,16 +630,28 @@ def retrieve_container(
api_version=api_version,
**kwargs,
)
# Decode container ID and extract provider info
original_container_id, resolved_custom_llm_provider, litellm_params = (
decode_managed_container_id_for_request(
container_id=container_id,
custom_llm_provider=custom_llm_provider,
litellm_params=litellm_params,
)
)
# True when input was a LiteLLM-managed ID (any length); needed to re-encode output for routing affinity
was_encoded = original_container_id != container_id
# get provider config
container_provider_config: Optional[
BaseContainerConfig
] = ProviderConfigManager.get_provider_container_config(
provider=litellm.LlmProviders(custom_llm_provider),
provider=litellm.LlmProviders(resolved_custom_llm_provider),
)
if container_provider_config is None:
raise ValueError(
f"Container provider config not found for provider: {custom_llm_provider}"
f"Container provider config not found for provider: {resolved_custom_llm_provider}"
)
# Pre Call logging
@ -635,14 +662,14 @@ def retrieve_container(
litellm_params={
"litellm_call_id": litellm_call_id,
},
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
)
# Set the correct call type
litellm_logging_obj.call_type = CallTypes.retrieve_container.value
return base_llm_http_handler.container_retrieve_handler(
container_id=container_id,
container_obj = base_llm_http_handler.container_retrieve_handler(
container_id=original_container_id, # Use decoded original ID
container_provider_config=container_provider_config,
litellm_params=litellm_params,
logging_obj=litellm_logging_obj,
@ -651,11 +678,33 @@ def retrieve_container(
timeout=timeout or DEFAULT_REQUEST_TIMEOUT,
_is_async=_is_async,
)
# Encode container_id with provider/model metadata for routing
# If input was encoded, preserve encoding in output using the decoded model_id
if isinstance(container_obj, ContainerObject):
# If input was encoded, use model_id from decoded params
litellm_metadata = kwargs.get("litellm_metadata", {})
if was_encoded and litellm_params.get("model_id"):
# Inject model_id from decoded container_id into litellm_metadata
if not litellm_metadata:
litellm_metadata = {}
if "model_info" not in litellm_metadata:
litellm_metadata["model_info"] = {}
litellm_metadata["model_info"]["id"] = litellm_params["model_id"]
container_obj = ContainerRequestUtils.encode_container_id_in_response(
response_obj=container_obj,
custom_llm_provider=resolved_custom_llm_provider,
litellm_metadata=litellm_metadata,
extra_body=None,
)
return container_obj
except Exception as e:
raise litellm.exception_type(
model="",
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
original_exception=e,
completion_kwargs=local_vars,
extra_kwargs=kwargs,
@ -667,7 +716,7 @@ def retrieve_container(
async def adelete_container(
container_id: str,
timeout=600, # default to 10 minutes
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -734,7 +783,7 @@ def delete_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
adelete_container: Literal[True],
**kwargs,
@ -749,7 +798,7 @@ def delete_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
adelete_container: Literal[False] = False,
**kwargs,
@ -766,7 +815,7 @@ def delete_container(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@ -780,6 +829,7 @@ def delete_container(
"""
local_vars = locals()
try:
resolved_custom_llm_provider: str = custom_llm_provider
litellm_logging_obj: LiteLLMLoggingObj = kwargs.pop("litellm_logging_obj") # type: ignore
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id")
_is_async = kwargs.pop("async_call", False) is True
@ -801,16 +851,28 @@ def delete_container(
api_version=api_version,
**kwargs,
)
# Decode container ID and extract provider info
original_container_id, resolved_custom_llm_provider, litellm_params = (
decode_managed_container_id_for_request(
container_id=container_id,
custom_llm_provider=custom_llm_provider,
litellm_params=litellm_params,
)
)
# True when input was a LiteLLM-managed ID (any length); needed to re-encode output for routing affinity
was_encoded = original_container_id != container_id
# get provider config
container_provider_config: Optional[
BaseContainerConfig
] = ProviderConfigManager.get_provider_container_config(
provider=litellm.LlmProviders(custom_llm_provider),
provider=litellm.LlmProviders(resolved_custom_llm_provider),
)
if container_provider_config is None:
raise ValueError(
f"Container provider config not found for provider: {custom_llm_provider}"
f"Container provider config not found for provider: {resolved_custom_llm_provider}"
)
# Pre Call logging
@ -821,14 +883,14 @@ def delete_container(
litellm_params={
"litellm_call_id": litellm_call_id,
},
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
)
# Set the correct call type
litellm_logging_obj.call_type = CallTypes.delete_container.value
return base_llm_http_handler.container_delete_handler(
container_id=container_id,
delete_result = base_llm_http_handler.container_delete_handler(
container_id=original_container_id, # Use decoded original ID
container_provider_config=container_provider_config,
litellm_params=litellm_params,
logging_obj=litellm_logging_obj,
@ -837,11 +899,33 @@ def delete_container(
timeout=timeout or DEFAULT_REQUEST_TIMEOUT,
_is_async=_is_async,
)
# Encode container_id in response with provider/model metadata for routing
# If input was encoded, preserve encoding in output using the decoded model_id
if isinstance(delete_result, DeleteContainerResult):
# If input was encoded, use model_id from decoded params
litellm_metadata = kwargs.get("litellm_metadata", {})
if was_encoded and litellm_params.get("model_id"):
# Inject model_id from decoded container_id into litellm_metadata
if not litellm_metadata:
litellm_metadata = {}
if "model_info" not in litellm_metadata:
litellm_metadata["model_info"] = {}
litellm_metadata["model_info"]["id"] = litellm_params["model_id"]
delete_result = ContainerRequestUtils.encode_container_id_in_response(
response_obj=delete_result,
custom_llm_provider=resolved_custom_llm_provider,
litellm_metadata=litellm_metadata,
extra_body=None,
)
return delete_result
except Exception as e:
raise litellm.exception_type(
model="",
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
original_exception=e,
completion_kwargs=local_vars,
extra_kwargs=kwargs,
@ -856,7 +940,7 @@ async def alist_container_files(
limit: Optional[int] = None,
order: Optional[str] = None,
timeout=600, # default to 10 minutes
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: Optional[Dict[str, Any]] = None,
extra_query: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,
@ -930,7 +1014,7 @@ def list_container_files(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
alist_container_files: Literal[True],
**kwargs,
@ -948,7 +1032,7 @@ def list_container_files(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
alist_container_files: Literal[False] = False,
**kwargs,
@ -968,7 +1052,7 @@ def list_container_files(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: Optional[Dict[str, Any]] = None,
extra_query: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,
@ -980,6 +1064,7 @@ def list_container_files(
"""
local_vars = locals()
try:
resolved_custom_llm_provider: str = custom_llm_provider
litellm_logging_obj: LiteLLMLoggingObj = kwargs.pop("litellm_logging_obj") # type: ignore
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id")
_is_async = kwargs.pop("async_call", False) is True
@ -1001,16 +1086,26 @@ def list_container_files(
api_version=api_version,
**kwargs,
)
# Decode container ID and extract provider info
original_container_id, resolved_custom_llm_provider, litellm_params = (
decode_managed_container_id_for_request(
container_id=container_id,
custom_llm_provider=custom_llm_provider,
litellm_params=litellm_params,
)
)
# get provider config
container_provider_config: Optional[
BaseContainerConfig
] = ProviderConfigManager.get_provider_container_config(
provider=litellm.LlmProviders(custom_llm_provider),
provider=litellm.LlmProviders(resolved_custom_llm_provider),
)
if container_provider_config is None:
raise ValueError(
f"Container provider config not found for provider: {custom_llm_provider}"
f"Container provider config not found for provider: {resolved_custom_llm_provider}"
)
# Pre Call logging
@ -1026,14 +1121,14 @@ def list_container_files(
litellm_params={
"litellm_call_id": litellm_call_id,
},
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
)
# Set the correct call type
litellm_logging_obj.call_type = CallTypes.list_container_files.value
return base_llm_http_handler.container_file_list_handler(
container_id=container_id,
container_id=original_container_id, # Use decoded original ID
container_provider_config=container_provider_config,
litellm_params=litellm_params,
logging_obj=litellm_logging_obj,
@ -1049,7 +1144,7 @@ def list_container_files(
except Exception as e:
raise litellm.exception_type(
model="",
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
original_exception=e,
completion_kwargs=local_vars,
extra_kwargs=kwargs,
@ -1062,7 +1157,7 @@ async def aupload_container_file(
container_id: str,
file: FileTypes,
timeout=600, # default to 10 minutes
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: Optional[Dict[str, Any]] = None,
extra_query: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,
@ -1151,7 +1246,7 @@ def upload_container_file(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
aupload_container_file: Literal[True],
**kwargs,
@ -1167,7 +1262,7 @@ def upload_container_file(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
*,
aupload_container_file: Literal[False] = False,
**kwargs,
@ -1185,7 +1280,7 @@ def upload_container_file(
api_key: Optional[str] = None,
api_base: Optional[str] = None,
api_version: Optional[str] = None,
custom_llm_provider: Literal["openai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: Optional[Dict[str, Any]] = None,
extra_query: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,
@ -1226,6 +1321,7 @@ def upload_container_file(
local_vars = locals()
try:
resolved_custom_llm_provider: str = custom_llm_provider
litellm_logging_obj: LiteLLMLoggingObj = kwargs.pop("litellm_logging_obj") # type: ignore
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id")
_is_async = kwargs.pop("async_call", False) is True
@ -1247,16 +1343,26 @@ def upload_container_file(
api_version=api_version,
**kwargs,
)
# Decode container ID and extract provider info
original_container_id, resolved_custom_llm_provider, litellm_params = (
decode_managed_container_id_for_request(
container_id=container_id,
custom_llm_provider=custom_llm_provider,
litellm_params=litellm_params,
)
)
# get provider config
container_provider_config: Optional[
BaseContainerConfig
] = ProviderConfigManager.get_provider_container_config(
provider=litellm.LlmProviders(custom_llm_provider),
provider=litellm.LlmProviders(resolved_custom_llm_provider),
)
if container_provider_config is None:
raise ValueError(
f"Container provider config not found for provider: {custom_llm_provider}"
f"Container provider config not found for provider: {resolved_custom_llm_provider}"
)
# Pre Call logging
@ -1267,7 +1373,7 @@ def upload_container_file(
litellm_params={
"litellm_call_id": litellm_call_id,
},
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
)
# Set the correct call type
@ -1282,14 +1388,14 @@ def upload_container_file(
extra_query=extra_query,
timeout=timeout or DEFAULT_REQUEST_TIMEOUT,
_is_async=_is_async,
container_id=container_id,
container_id=original_container_id, # Use decoded original ID
file=file,
)
except Exception as e:
raise litellm.exception_type(
model="",
custom_llm_provider=custom_llm_provider,
custom_llm_provider=resolved_custom_llm_provider,
original_exception=e,
completion_kwargs=local_vars,
extra_kwargs=kwargs,

93
litellm/containers/utils.py
View file

@ -1,10 +1,38 @@
from typing import Dict
from typing import Any, Dict, Optional, TypeVar
from litellm.llms.base_llm.containers.transformation import BaseContainerConfig
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.types.containers.main import (
ContainerCreateOptionalRequestParams,
ContainerListOptionalRequestParams,
)
from litellm.types.router import GenericLiteLLMParams
def decode_managed_container_id_for_request(
container_id: str,
custom_llm_provider: str,
litellm_params: GenericLiteLLMParams,
) -> tuple[str, str, GenericLiteLLMParams]:
"""Decode a LiteLLM-managed container ID for upstream API calls.
Returns:
(original_container_id, resolved_provider, updated_litellm_params)
"""
decoded = ResponsesAPIRequestUtils._decode_container_id(container_id)
original_container_id = decoded.get("response_id", container_id)
decoded_provider = decoded.get("custom_llm_provider")
if decoded_provider and custom_llm_provider == "openai":
custom_llm_provider = decoded_provider
decoded_model_id = decoded.get("model_id")
if decoded_model_id and not litellm_params.get("model_id"):
litellm_params["model_id"] = decoded_model_id
return original_container_id, custom_llm_provider, litellm_params
T = TypeVar("T")
class ContainerRequestUtils:
@ -68,3 +96,66 @@ class ContainerRequestUtils:
container_list_optional_params[param] = passed_params[param] # type: ignore
return container_list_optional_params
@staticmethod
def encode_container_id_in_response(
response_obj: T,
custom_llm_provider: Optional[str],
litellm_metadata: Optional[Dict[str, Any]] = None,
extra_body: Optional[Dict[str, Any]] = None,
) -> T:
"""
Encode container_id in response object with provider/model metadata for routing.
This mirrors the responses API pattern where response IDs are encoded with
routing metadata so follow-up calls can route to the correct provider.
Encodes when:
1. litellm_metadata contains model_info.id (indicating router/proxy usage), OR
2. extra_body contains target_model_names (indicating model-specific routing)
Direct SDK calls with explicit custom_llm_provider and no routing hints return raw IDs.
Args:
response_obj: Response object with an `id` attribute (ContainerObject, DeleteContainerResult, etc.)
custom_llm_provider: Provider name (e.g., "azure", "openai")
litellm_metadata: Optional litellm_metadata dict that may contain model_info.id
extra_body: Optional extra_body dict that may contain target_model_names
Returns:
The same response object with encoded container_id (if routing metadata present)
"""
# Extract model_id from litellm_metadata
litellm_metadata = litellm_metadata or {}
model_info: Dict[str, Any] = litellm_metadata.get("model_info", {}) or {}
model_id = model_info.get("id")
# Check if we should encode based on routing metadata
should_encode = False
# Case 1: Router/proxy usage (model_id from router)
if model_id is not None:
should_encode = True
# Case 2: target_model_names in extra_body (model-specific routing)
if extra_body and "target_model_names" in extra_body:
should_encode = True
# Extract model_id from target_model_names if not already set
if model_id is None:
target_models = extra_body["target_model_names"]
# Use first model as model_id for encoding
if isinstance(target_models, str):
model_id = target_models.split(",")[0].strip()
elif isinstance(target_models, list) and len(target_models) > 0:
model_id = str(target_models[0]).strip()
# Only encode if we have routing metadata
if should_encode and response_obj and hasattr(response_obj, "id"):
encoded_id = ResponsesAPIRequestUtils._build_container_id(
custom_llm_provider=custom_llm_provider,
model_id=model_id,
container_id=response_obj.id,
)
response_obj.id = encoded_id
return response_obj

150
litellm/files/main.py
View file

@ -10,7 +10,7 @@ import contextvars
import time
import uuid as uuid_module
from functools import partial
from typing import Any, Coroutine, Dict, Literal, Optional, Union, cast
from typing import Any,Coroutine, Dict, Literal, Optional, Union, cast
import httpx
@ -30,12 +30,10 @@ FileRetrieveProvider = Literal[
]
FileDeleteProvider = Literal["openai", "azure", "gemini", "manus", "anthropic"]
FileListProvider = Literal["openai", "azure", "manus", "anthropic"]
FileContentProvider = Literal[
"openai", "azure", "vertex_ai", "bedrock", "hosted_vllm", "anthropic", "manus"
]
import litellm
from litellm import get_secret_str
from litellm.files.streaming import FileContentStreamingResponse
from litellm.files.types import FileContentProvider, FileContentStreamingResult
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.common_utils import get_azure_credentials
@ -55,10 +53,7 @@ from litellm.types.llms.openai import (
OpenAIFileObject,
)
from litellm.types.router import *
from litellm.types.utils import (
OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS,
LlmProviders,
)
from litellm.types.utils import OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS, LlmProviders
from litellm.utils import (
ProviderConfigManager,
client,
@ -69,6 +64,15 @@ from litellm.utils import (
base_llm_http_handler = BaseLLMHTTPHandler()
####### ENVIRONMENT VARIABLES ###################
def _should_sdk_support_streaming(
custom_llm_provider: Optional[Union[FileContentProvider, str]],
) -> bool:
"""
Return whether file content streaming is supported for the provider.
"""
return custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS
openai_files_instance = OpenAIFilesAPI()
azure_files_instance = AzureOpenAIFilesAPI()
vertex_ai_files_instance = VertexAIFilesHandler()
@ -772,8 +776,10 @@ async def afile_content(
custom_llm_provider: FileContentProvider = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
chunk_size: int = 1024 * 1024,
stream: bool = False,
**kwargs,
) -> HttpxBinaryResponseContent:
) -> Union[HttpxBinaryResponseContent, FileContentStreamingResult]:
"""
Async: Get file contents
@ -787,11 +793,13 @@ async def afile_content(
# Use a partial function to pass your keyword arguments
func = partial(
file_content,
file_id,
model,
custom_llm_provider,
extra_headers,
extra_body,
file_id=file_id,
model=model,
custom_llm_provider=custom_llm_provider,
extra_headers=extra_headers,
extra_body=extra_body,
chunk_size=chunk_size,
stream=stream,
**kwargs,
)
@ -816,8 +824,15 @@ def file_content(
custom_llm_provider: Optional[Union[FileContentProvider, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
chunk_size: int = 1024 * 1024,
stream: bool = False,
**kwargs,
) -> Union[HttpxBinaryResponseContent, Coroutine[Any, Any, HttpxBinaryResponseContent]]:
) -> Union[
HttpxBinaryResponseContent,
FileContentStreamingResult,
Coroutine[Any, Any, HttpxBinaryResponseContent],
Coroutine[Any, Any, FileContentStreamingResult],
]:
"""
Returns the contents of the specified file.
@ -859,6 +874,23 @@ def file_content(
_is_async = kwargs.pop("afile_content", False) is True
if stream and _should_sdk_support_streaming(custom_llm_provider):
return file_content_streaming(
file_id=file_id,
model=model,
custom_llm_provider=custom_llm_provider,
extra_headers=extra_headers,
extra_body=extra_body,
chunk_size=chunk_size,
optional_params=optional_params,
timeout=timeout,
logging_obj=cast(
Optional[LiteLLMLoggingObj], kwargs.get("litellm_logging_obj")
),
_is_async=_is_async,
client=client,
)
# Check if provider has a custom files config (e.g., Anthropic, Manus)
provider_config = ProviderConfigManager.get_provider_files_config(
model="",
@ -982,3 +1014,89 @@ def file_content(
return response
except Exception as e:
raise e
def file_content_streaming(
*,
file_id: str,
model: Optional[str],
custom_llm_provider: Optional[Union[FileContentProvider, str]],
extra_headers: Optional[Dict[str, str]],
extra_body: Optional[Dict[str, str]],
chunk_size: int,
optional_params: GenericLiteLLMParams,
timeout: Union[float, httpx.Timeout],
logging_obj: Optional[LiteLLMLoggingObj],
_is_async: bool,
client: Optional[Any],
) -> Union[FileContentStreamingResult, Coroutine[Any, Any, FileContentStreamingResult]]:
if logging_obj is not None:
logging_obj.model = model or ""
logging_obj.model_call_details["model"] = model or ""
logging_obj.model_call_details["custom_llm_provider"] = custom_llm_provider
litellm_params = logging_obj.model_call_details.get("litellm_params", {}) or {}
if optional_params.api_base is not None:
litellm_params["api_base"] = optional_params.api_base
logging_obj.model_call_details["litellm_params"] = litellm_params
def _wrap_streaming_result(
response: FileContentStreamingResult,
) -> FileContentStreamingResult:
return FileContentStreamingResult(
stream_iterator=FileContentStreamingResponse(
stream_iterator=response.stream_iterator,
file_id=file_id,
model=model,
custom_llm_provider=custom_llm_provider,
logging_obj=logging_obj,
),
headers=response.headers,
)
response: Union[
FileContentStreamingResult, Coroutine[Any, Any, FileContentStreamingResult]
] = FileContentStreamingResult(stream_iterator=iter(()), headers={})
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
openai_creds = get_openai_credentials(
api_base=optional_params.api_base,
api_key=optional_params.api_key,
organization=optional_params.organization,
)
response = openai_files_instance.file_content_streaming(
_is_async=_is_async,
file_content_request=FileContentRequest(
file_id=file_id,
extra_headers=extra_headers,
extra_body=extra_body,
),
api_base=openai_creds.api_base,
api_key=openai_creds.api_key,
timeout=timeout,
max_retries=optional_params.max_retries,
organization=openai_creds.organization,
chunk_size=chunk_size,
client=client,
)
else:
raise litellm.exceptions.BadRequestError(
message="LiteLLM doesn't support {} for streaming 'file_content'. Supported providers are {}.".format(
custom_llm_provider,
sorted(OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS),
),
model="n/a",
llm_provider=custom_llm_provider,
response=httpx.Response(
status_code=400,
content="Unsupported provider",
request=httpx.Request(method="create_thread", url="https://github.com/BerriAI/litellm"), # type: ignore
),
)
if asyncio.iscoroutine(response):
async def _await_and_wrap() -> FileContentStreamingResult:
return _wrap_streaming_result(await response)
return _await_and_wrap()
return _wrap_streaming_result(response)

236
litellm/files/streaming.py Normal file
View file

@ -0,0 +1,236 @@
import datetime
import traceback
from typing import TYPE_CHECKING, Any, AsyncIterator, Dict, Iterator, Optional, Union, cast
import anyio
from litellm.files.types import FileContentProvider
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import (
Logging as LiteLLMLoggingObj,
)
from litellm.types.utils import StandardLoggingHiddenParams, StandardLoggingPayload
class FileContentStreamingResponse:
"""
Iterator wrapper for file content streaming that carries LiteLLM metadata
and emits success/failure callbacks once the stream finishes.
"""
def __init__(
self,
stream_iterator: Union[Iterator[bytes], AsyncIterator[bytes]],
file_id: str,
model: Optional[str],
custom_llm_provider: Optional[Union[FileContentProvider, str]],
logging_obj: Optional["LiteLLMLoggingObj"],
) -> None:
self.stream_iterator = stream_iterator
self.file_id = file_id
self.model = model
self.custom_llm_provider = custom_llm_provider
self.logging_obj = logging_obj
self.standard_logging_object: Optional["StandardLoggingPayload"] = None
self._hidden_params: Dict[str, Any] = {}
self._logging_completed = False
self._close_completed = False
self._start_time = (
logging_obj.start_time
if logging_obj is not None and getattr(logging_obj, "start_time", None)
else datetime.datetime.now()
)
self._sync_hidden_params()
def __iter__(self) -> "FileContentStreamingResponse":
if not hasattr(self.stream_iterator, "__next__"):
raise TypeError("File content stream does not support sync iteration")
return self
def __next__(self) -> bytes:
if not hasattr(self.stream_iterator, "__next__"):
raise TypeError("File content stream does not support sync iteration")
try:
return next(cast(Iterator[bytes], self.stream_iterator))
except StopIteration:
self._log_success_sync()
raise
except Exception as e:
self._log_failure_sync(e)
raise
def __aiter__(self) -> "FileContentStreamingResponse":
if not hasattr(self.stream_iterator, "__anext__"):
raise TypeError("File content stream does not support async iteration")
return self
async def __anext__(self) -> bytes:
if not hasattr(self.stream_iterator, "__anext__"):
raise TypeError("File content stream does not support async iteration")
try:
return await cast(AsyncIterator[bytes], self.stream_iterator).__anext__()
except StopAsyncIteration:
await self._log_success_async()
raise
except Exception as e:
await self._log_failure_async(e)
raise
async def aclose(self) -> None:
if self._close_completed:
return
self._close_completed = True
self._logging_completed = True
stream_to_close = self.stream_iterator
self.stream_iterator = cast(Union[Iterator[bytes], AsyncIterator[bytes]], iter(()))
# Shield cleanup from request cancellation so upstream HTTP connections
# are released promptly on client disconnects.
with anyio.CancelScope(shield=True):
if hasattr(stream_to_close, "aclose"):
await cast(AsyncIterator[bytes], stream_to_close).aclose() # type: ignore[attr-defined]
elif hasattr(stream_to_close, "close"):
result = cast(Iterator[bytes], stream_to_close).close() # type: ignore[attr-defined]
if result is not None:
await result
def close(self) -> None:
if self._close_completed:
return
self._close_completed = True
self._logging_completed = True
stream_to_close = self.stream_iterator
self.stream_iterator = cast(Union[Iterator[bytes], AsyncIterator[bytes]], iter(()))
if hasattr(stream_to_close, "close"):
cast(Iterator[bytes], stream_to_close).close() # type: ignore[attr-defined]
def _build_logging_response(self) -> Dict[str, str]:
response = {
"id": self.file_id,
"object": "file.content",
}
if self.model:
response["model"] = self.model
return response
def _sync_hidden_params(self) -> None:
litellm_params: dict[str, Any] = {}
if self.logging_obj is not None:
litellm_params = (
self.logging_obj.model_call_details.get("litellm_params", {}) or {}
)
if "api_base" not in self._hidden_params and litellm_params.get("api_base"):
self._hidden_params["api_base"] = litellm_params["api_base"]
# The generic client decorator infers `model` from the first positional arg,
# which is `file_id` for this API. Correct it before logging callbacks run.
self._hidden_params["litellm_model_name"] = self.model
if "response_cost" not in self._hidden_params:
self._hidden_params["response_cost"] = None
def _build_standard_logging_object(
self,
end_time: datetime.datetime,
) -> Optional["StandardLoggingPayload"]:
if self.standard_logging_object is not None:
return self.standard_logging_object
if self.logging_obj is None:
return None
from litellm.litellm_core_utils.litellm_logging import (
get_standard_logging_object_payload,
)
self._sync_hidden_params()
payload = get_standard_logging_object_payload(
kwargs=self.logging_obj.model_call_details,
init_response_obj=self._build_logging_response(),
start_time=self._start_time,
end_time=end_time,
logging_obj=self.logging_obj,
status="success",
)
if payload is None:
return None
merged_hidden_params = cast(
"StandardLoggingHiddenParams",
{
**cast(Dict[str, Any], payload.get("hidden_params") or {}),
**self._hidden_params,
},
)
payload["hidden_params"] = merged_hidden_params
payload["response"] = self._build_logging_response()
if self.custom_llm_provider is not None:
payload["custom_llm_provider"] = self.custom_llm_provider
if self.model is not None:
payload["model"] = self.model
if self._hidden_params.get("api_base"):
payload["api_base"] = cast(str, self._hidden_params["api_base"])
self.standard_logging_object = payload
return payload
async def _log_success_async(self) -> None:
if self._logging_completed or self.logging_obj is None:
return
self._logging_completed = True
end_time = datetime.datetime.now()
standard_logging_object = self._build_standard_logging_object(end_time=end_time)
await self.logging_obj.async_success_handler(
result=self._build_logging_response(),
start_time=self._start_time,
end_time=end_time,
standard_logging_object=standard_logging_object,
)
self.logging_obj.handle_sync_success_callbacks_for_async_calls(
result=self._build_logging_response(),
start_time=self._start_time,
end_time=end_time,
)
def _log_success_sync(self) -> None:
if self._logging_completed or self.logging_obj is None:
return
self._logging_completed = True
end_time = datetime.datetime.now()
standard_logging_object = self._build_standard_logging_object(end_time=end_time)
self.logging_obj.success_handler(
result=self._build_logging_response(),
start_time=self._start_time,
end_time=end_time,
standard_logging_object=standard_logging_object,
)
async def _log_failure_async(self, error: Exception) -> None:
if self._logging_completed or self.logging_obj is None:
return
self._logging_completed = True
end_time = datetime.datetime.now()
traceback_str = traceback.format_exc()
self.logging_obj.failure_handler(
error, traceback_str, self._start_time, end_time
)
await self.logging_obj.async_failure_handler(
error, traceback_str, self._start_time, end_time
)
def _log_failure_sync(self, error: Exception) -> None:
if self._logging_completed or self.logging_obj is None:
return
self._logging_completed = True
end_time = datetime.datetime.now()
self.logging_obj.failure_handler(
error, traceback.format_exc(), self._start_time, end_time
)

11
litellm/files/types.py Normal file
View file

@ -0,0 +1,11 @@
from typing import AsyncIterator, Dict, Iterator, Literal, NamedTuple, Union
FileContentProvider = Literal[
"openai", "azure", "vertex_ai", "bedrock", "hosted_vllm", "anthropic", "manus"
]
class FileContentStreamingResult(NamedTuple):
stream_iterator: Union[Iterator[bytes], AsyncIterator[bytes]]
headers: Dict[str, str]

8
litellm/integrations/dotprompt/prompt_manager.py
View file

@ -7,7 +7,8 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Union
import yaml
from jinja2 import DictLoader, Environment, select_autoescape
from jinja2 import DictLoader, select_autoescape
from jinja2.sandbox import ImmutableSandboxedEnvironment
class PromptTemplate:
@ -59,7 +60,10 @@ class PromptManager:
self.prompt_directory = Path(prompt_directory) if prompt_directory else None
self.prompts: Dict[str, PromptTemplate] = {}
self.prompt_file = prompt_file
self.jinja_env = Environment(
# Sandboxed env: templates can come from user input via /prompts/test,
# so we must block access to unsafe Python attributes and mutation of
# caller-supplied mutables.
self.jinja_env = ImmutableSandboxedEnvironment(
loader=DictLoader({}),
autoescape=select_autoescape(["html", "xml"]),
# Use Handlebars-style delimiters to match Dotprompt spec

24
litellm/integrations/prometheus.py
View file

@ -86,6 +86,11 @@ class PrometheusLogger(CustomLogger):
# Always initialize label_filters, even for non-premium users
self.label_filters = self._parse_prometheus_config()
_custom_buckets = litellm.prometheus_latency_buckets
self.latency_buckets = (
tuple(_custom_buckets) if _custom_buckets is not None else LATENCY_BUCKETS
)
# Create metric factory functions
self._counter_factory = self._create_metric_factory(Counter)
self._gauge_factory = self._create_metric_factory(Gauge)
@ -114,14 +119,14 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric(
"litellm_request_total_latency_metric"
),
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
self.litellm_llm_api_latency_metric = self._histogram_factory(
"litellm_llm_api_latency_metric",
"Total latency (seconds) for a models LLM API call",
labelnames=self.get_labels_for_metric("litellm_llm_api_latency_metric"),
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
self.litellm_llm_api_time_to_first_token_metric = self._histogram_factory(
@ -137,7 +142,7 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric(
"litellm_llm_api_time_to_first_token_metric"
),
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
# Counter for spend
@ -314,7 +319,7 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric(
"litellm_overhead_latency_metric"
),
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
# Request queue time metric
@ -324,7 +329,7 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric(
"litellm_request_queue_time_seconds"
),
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
# Guardrail metrics
@ -332,7 +337,7 @@ class PrometheusLogger(CustomLogger):
"litellm_guardrail_latency_seconds",
"Latency (seconds) for guardrail execution",
labelnames=["guardrail_name", "status", "error_type", "hook_type"],
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
self.litellm_guardrail_errors_total = self._counter_factory(
@ -1031,6 +1036,9 @@ class PrometheusLogger(CustomLogger):
user_api_key_org_id = standard_logging_payload["metadata"].get(
"user_api_key_org_id"
)
user_api_key_org_alias = standard_logging_payload["metadata"].get(
"user_api_key_org_alias"
)
output_tokens = standard_logging_payload["completion_tokens"]
tokens_used = standard_logging_payload["total_tokens"]
response_cost = standard_logging_payload["response_cost"]
@ -1068,6 +1076,8 @@ class PrometheusLogger(CustomLogger):
model_group=standard_logging_payload["model_group"],
team=user_api_team,
team_alias=user_api_team_alias,
org_id=user_api_key_org_id,
org_alias=user_api_key_org_alias,
user=user_id,
user_email=standard_logging_payload["metadata"]["user_api_key_user_email"],
status_code="200",
@ -1746,6 +1756,8 @@ class PrometheusLogger(CustomLogger):
api_key_alias=user_api_key_dict.key_alias,
team=user_api_key_dict.team_id,
team_alias=user_api_key_dict.team_alias,
org_id=user_api_key_dict.org_id,
org_alias=user_api_key_dict.organization_alias,
requested_model=request_data.get("model", ""),
status_code=str(status_code),
exception_status=str(status_code),

8
litellm/integrations/prometheus_services.py
View file

@ -5,6 +5,7 @@
from typing import Dict, List, Optional, Union
import litellm
from litellm._logging import print_verbose, verbose_logger
from litellm.types.integrations.prometheus import LATENCY_BUCKETS
from litellm.types.services import (
@ -35,6 +36,11 @@ class PrometheusServicesLogger:
"Missing prometheus_client. Run `pip install prometheus-client`"
)
_custom_buckets = litellm.prometheus_latency_buckets
self.latency_buckets = (
tuple(_custom_buckets) if _custom_buckets is not None else LATENCY_BUCKETS
)
self.Histogram = Histogram
self.Counter = Counter
self.Gauge = Gauge
@ -130,7 +136,7 @@ class PrometheusServicesLogger:
metric_name,
"Latency for {} service".format(service),
labelnames=[service],
buckets=LATENCY_BUCKETS,
buckets=self.latency_buckets,
)
def create_gauge(self, service: str, type_of_request: str):

43
litellm/integrations/s3_v2.py
View file

@ -7,6 +7,7 @@ NOTE 1: S3 does not provide a BATCH PUT API endpoint, so we create tasks to uplo
"""
import asyncio
import time
from datetime import datetime
from typing import List, Optional, cast
@ -403,11 +404,23 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
# Prepare the signed headers
signed_headers = dict(aws_request.headers.items())
# Make the request
response = await self.async_httpx_client.put(
url, data=json_string, headers=signed_headers
)
response.raise_for_status()
# Make the request with retry for transient S3 errors (500/503)
max_retries = 3
for attempt in range(max_retries):
response = await self.async_httpx_client.put(
url, data=json_string, headers=signed_headers
)
if response.status_code in (500, 503) and attempt < max_retries - 1:
wait_time = 2**attempt # 1s, 2s
verbose_logger.warning(
f"S3 upload returned {response.status_code}, retrying in {wait_time}s "
f"(attempt {attempt + 1}/{max_retries}) "
f"key={batch_logging_element.s3_object_key}"
)
await asyncio.sleep(wait_time)
continue
response.raise_for_status()
break
except Exception as e:
verbose_logger.exception(f"Error uploading to s3: {str(e)}")
self.handle_callback_failure(callback_name="S3Logger")
@ -582,9 +595,23 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
if self.s3_verify is not None
else None
)
# Make the request
response = httpx_client.put(url, data=json_string, headers=signed_headers)
response.raise_for_status()
# Make the request with retry for transient S3 errors (500/503)
max_retries = 3
for attempt in range(max_retries):
response = httpx_client.put(
url, data=json_string, headers=signed_headers
)
if response.status_code in (500, 503) and attempt < max_retries - 1:
wait_time = 2**attempt # 1s, 2s
verbose_logger.warning(
f"S3 upload returned {response.status_code}, retrying in {wait_time}s "
f"(attempt {attempt + 1}/{max_retries}) "
f"key={batch_logging_element.s3_object_key}"
)
time.sleep(wait_time)
continue
response.raise_for_status()
break
except Exception as e:
verbose_logger.exception(f"Error uploading to s3: {str(e)}")
self.handle_callback_failure(callback_name="S3Logger")

12
litellm/integrations/websearch_interception/handler.py
View file

@ -230,8 +230,15 @@ class WebSearchInterceptionLogger(CustomLogger):
# Keep other tools as-is
converted_tools.append(tool)
# Update tools in-place and return full kwargs
kwargs["tools"] = converted_tools
if kwargs.get("stream"):
verbose_logger.debug(
"WebSearchInterception: deployment hook converting stream=True to stream=False"
)
kwargs["stream"] = False
kwargs["_websearch_interception_converted_stream"] = True
return kwargs
@classmethod
@ -344,13 +351,12 @@ class WebSearchInterceptionLogger(CustomLogger):
else:
converted_tools.append(tool)
# Update kwargs with converted tools
kwargs["tools"] = converted_tools
verbose_logger.debug(
f"WebSearchInterception: Tools after conversion: {[t.get('name') for t in converted_tools]}"
)
# Convert stream=True to stream=False for WebSearch interception
# Also convert here for direct callers that bypass the deployment hook.
if kwargs.get("stream"):
verbose_logger.debug(
"WebSearchInterception: Converting stream=True to stream=False"

12
litellm/litellm_core_utils/litellm_logging.py
View file

@ -613,7 +613,17 @@ class Logging(LiteLLMLoggingBaseClass):
base_litellm_params["metadata"] = kwargs["litellm_metadata"].copy()
if litellm_params:
# Merge metadata carefully — don't overwrite the merged metadata
# from kwargs/litellm_metadata with the caller's litellm_params metadata.
# e.g. anthropic_messages passes Anthropic's native metadata ({user_id: ...})
# in litellm_params, which would overwrite proxy key-auth fields.
lp_metadata = litellm_params.pop("metadata", None)
base_litellm_params.update(litellm_params)
if lp_metadata and isinstance(lp_metadata, dict):
base_litellm_params.setdefault("metadata", {})
for k, v in lp_metadata.items():
if k not in base_litellm_params["metadata"]:
base_litellm_params["metadata"][k] = v
self.update_environment_variables(
litellm_params=base_litellm_params,
@ -4754,6 +4764,7 @@ class StandardLoggingPayloadSetup:
user_api_key_budget_reset_at=None,
user_api_key_team_id=None,
user_api_key_org_id=None,
user_api_key_org_alias=None,
user_api_key_project_id=None,
user_api_key_project_alias=None,
user_api_key_user_id=None,
@ -5586,6 +5597,7 @@ def get_standard_logging_metadata(
user_api_key_budget_reset_at=None,
user_api_key_team_id=None,
user_api_key_org_id=None,
user_api_key_org_alias=None,
user_api_key_project_id=None,
user_api_key_project_alias=None,
user_api_key_user_id=None,

12
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
View file

@ -322,9 +322,8 @@ class StandardBuiltInToolCostTracking:
)
if has_url_citations:
return True
# Fallback: Check usage object for providers that use usage instead of annotations
# (e.g., Vertex AI Gemini uses usage.prompt_tokens_details.web_search_requests)
if usage is not None:
# Vertex AI Gemini uses usage.prompt_tokens_details.web_search_requests
if (
hasattr(usage, "prompt_tokens_details")
and usage.prompt_tokens_details is not None
@ -335,6 +334,15 @@ class StandardBuiltInToolCostTracking:
and usage.prompt_tokens_details.web_search_requests is not None
):
return True
# Anthropic Claude (direct API and Vertex AI) uses server_tool_use.web_search_requests.
# Without this check, Claude ModelResponse always falls through to return False
# and _handle_web_search_cost() is never called.
if (
hasattr(usage, "server_tool_use")
and usage.server_tool_use is not None
and usage.server_tool_use.web_search_requests is not None
):
return True
return False
elif isinstance(response_object, ResponsesAPIResponse):
# response api explicitly includes web_search_call in the output

18
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -4371,17 +4371,19 @@ class BedrockConverseMessagesProcessor:
# if initial message is assistant message
if messages[0].get("role") is not None and messages[0]["role"] == "assistant":
if user_continue_message is not None:
messages.insert(0, user_continue_message)
elif litellm.modify_params:
messages.insert(0, DEFAULT_USER_CONTINUE_MESSAGE)
if not messages[0].get("prefix"):
if user_continue_message is not None:
messages.insert(0, user_continue_message)
elif litellm.modify_params:
messages.insert(0, DEFAULT_USER_CONTINUE_MESSAGE)
# if final message is assistant message
if messages[-1].get("role") is not None and messages[-1]["role"] == "assistant":
if user_continue_message is not None:
messages.append(user_continue_message)
elif litellm.modify_params:
messages.append(DEFAULT_USER_CONTINUE_MESSAGE)
if not messages[-1].get("prefix"):
if user_continue_message is not None:
messages.append(user_continue_message)
elif litellm.modify_params:
messages.append(DEFAULT_USER_CONTINUE_MESSAGE)
return messages
@staticmethod

7
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -123,10 +123,13 @@ class ChunkProcessor:
finish_reason = "stop"
for chunk in chunks:
if "choices" in chunk and len(chunk["choices"]) > 0:
chunk_finish_reason = None
if hasattr(chunk["choices"][0], "finish_reason"):
finish_reason = chunk["choices"][0].finish_reason
chunk_finish_reason = chunk["choices"][0].finish_reason
elif "finish_reason" in chunk["choices"][0]:
finish_reason = chunk["choices"][0]["finish_reason"]
chunk_finish_reason = chunk["choices"][0]["finish_reason"]
if chunk_finish_reason is not None:
finish_reason = chunk_finish_reason
# Initialize the response dictionary
response = ModelResponse(

6
litellm/litellm_core_utils/streaming_handler.py
View file

@ -1134,7 +1134,11 @@ class CustomStreamWrapper:
):
if self.received_finish_reason is not None:
_chunk_has_content = isinstance(chunk, dict) and (
bool(chunk.get("text", "")) or chunk.get("tool_use") is not None
bool(chunk.get("text", ""))
or chunk.get("tool_use") is not None
# Usage-only final chunks are valid and needed to surface
# finish_reason/usage to downstream translators.
or chunk.get("usage") is not None
)
if not _chunk_has_content and (
not isinstance(chunk, dict)

12
litellm/llms/__init__.py
View file

@ -34,6 +34,18 @@ def get_cost_for_web_search_request(
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
elif custom_llm_provider.startswith("vertex_ai"):
# Anthropic Claude models on Vertex AI populate server_tool_use.web_search_requests
# (same as the direct Anthropic API), not prompt_tokens_details.web_search_requests
# (which is the Gemini field). Route claude-* models to the Anthropic calculator.
model_key: str = model_info.get("key", "") if model_info else ""
if "claude" in model_key.lower():
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
verbose_logger.debug(
"vertex_ai/claude model detected — routing web search cost to Anthropic calculator"
)
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
from .vertex_ai.gemini.cost_calculator import (
cost_per_web_search_request as cost_per_web_search_request_vertex_ai,
)

19
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -21,6 +21,10 @@ from litellm.llms.anthropic.experimental_pass_through.adapters.transformation im
LiteLLMAnthropicMessagesAdapter,
)
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.llms.base_llm.guardrail_translation.utils import (
effective_skip_system_message_for_guardrail,
openai_messages_without_system,
)
from litellm.proxy.pass_through_endpoints.llm_provider_handlers.anthropic_passthrough_logging_handler import (
AnthropicPassthroughLoggingHandler,
)
@ -29,6 +33,7 @@ from litellm.types.llms.anthropic import (
AnthropicMessagesRequest,
)
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionToolCallChunk,
ChatCompletionToolParam,
)
@ -75,6 +80,8 @@ class AnthropicMessagesHandler(BaseTranslation):
if messages is None:
return data
skip_system = effective_skip_system_message_for_guardrail(guardrail_to_apply)
(
chat_completion_compatible_request,
_tool_name_mapping,
@ -83,7 +90,12 @@ class AnthropicMessagesHandler(BaseTranslation):
anthropic_message_request=cast(AnthropicMessagesRequest, data.copy())
)
structured_messages = chat_completion_compatible_request.get("messages", [])
structured_messages = cast(
List[AllMessageValues],
chat_completion_compatible_request.get("messages", []),
)
if skip_system:
structured_messages = openai_messages_without_system(structured_messages)
texts_to_check: List[str] = []
images_to_check: List[str] = []
@ -102,6 +114,7 @@ class AnthropicMessagesHandler(BaseTranslation):
texts_to_check=texts_to_check,
images_to_check=images_to_check,
task_mappings=task_mappings,
skip_system_message=skip_system,
)
# Step 2: Apply guardrail to all texts in batch
@ -165,12 +178,16 @@ class AnthropicMessagesHandler(BaseTranslation):
texts_to_check: List[str],
images_to_check: List[str],
task_mappings: List[Tuple[int, Optional[int]]],
skip_system_message: bool = False,
) -> None:
"""
Extract text content and images from a message.
Override this method to customize text/image extraction logic.
"""
if skip_system_message and str(message.get("role") or "").lower() == "system":
return
content = message.get("content", None)
tools = message.get("tools", None)
if content is None and tools is None:

21
litellm/llms/anthropic/chat/handler.py
View file

@ -89,7 +89,12 @@ async def make_call(
try:
response = await client.post(
api_base, headers=headers, data=data, stream=True, timeout=timeout
api_base,
headers=headers,
data=data,
stream=True,
timeout=timeout,
logging_obj=logging_obj,
)
except httpx.HTTPStatusError as e:
error_headers = getattr(e, "headers", None)
@ -142,7 +147,12 @@ def make_sync_call(
try:
response = client.post(
api_base, headers=headers, data=data, stream=True, timeout=timeout
api_base,
headers=headers,
data=data,
stream=True,
timeout=timeout,
logging_obj=logging_obj,
)
except httpx.HTTPStatusError as e:
error_headers = getattr(e, "headers", None)
@ -266,7 +276,11 @@ class AnthropicChatCompletion(BaseLLM):
try:
response = await async_handler.post(
api_base, headers=headers, json=data, timeout=timeout
api_base,
headers=headers,
json=data,
timeout=timeout,
logging_obj=logging_obj,
)
except Exception as e:
## LOGGING
@ -469,6 +483,7 @@ class AnthropicChatCompletion(BaseLLM):
headers=headers,
data=json.dumps(data),
timeout=timeout,
logging_obj=logging_obj,
)
except Exception as e:
status_code = getattr(e, "status_code", 500)

78
litellm/llms/anthropic/chat/transformation.py
View file

@ -19,6 +19,7 @@ from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.llms.base_llm.base_utils import type_to_response_format_param
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.types.llms.anthropic import (
ANTHROPIC_ADVISOR_TOOL_TYPE,
ANTHROPIC_BETA_HEADER_VALUES,
ANTHROPIC_HOSTED_TOOLS,
AllAnthropicMessageValues,
@ -75,7 +76,12 @@ from litellm.utils import (
token_counter,
)
from ..common_utils import AnthropicError, AnthropicModelInfo, process_anthropic_headers
from ..common_utils import (
AnthropicError,
AnthropicModelInfo,
process_anthropic_headers,
strip_advisor_blocks_from_messages,
)
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -508,6 +514,23 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
type="tool_search_tool_bm25_20251119",
name=tool_name,
)
elif tool["type"] == ANTHROPIC_ADVISOR_TOOL_TYPE:
from litellm.types.llms.anthropic import AnthropicAdvisorTool
_tool_dict = cast(dict, tool)
advisor_model = _tool_dict.get("model")
if not isinstance(advisor_model, str):
raise ValueError("Advisor tool must have a valid model")
_advisor_tool = AnthropicAdvisorTool(
type=ANTHROPIC_ADVISOR_TOOL_TYPE,
name="advisor",
model=advisor_model,
)
if _tool_dict.get("max_uses") is not None:
_advisor_tool["max_uses"] = _tool_dict["max_uses"]
if _tool_dict.get("caching") is not None:
_advisor_tool["caching"] = _tool_dict["caching"]
returned_tool = _advisor_tool # type: ignore[assignment]
if returned_tool is None and mcp_server is None:
raise ValueError(f"Unsupported tool type: {tool['type']}")
@ -1311,6 +1334,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.FAST_MODE_2026_02_01.value
)
for tool in _tools:
if tool.get("type") == ANTHROPIC_ADVISOR_TOOL_TYPE:
self._ensure_beta_header(
headers, ANTHROPIC_BETA_HEADER_VALUES.ADVISOR_TOOL_2026_03_01.value
)
break
return headers
def transform_request(
@ -1390,6 +1419,16 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
message="{}\nReceived Messages={}".format(str(e), messages),
) # don't use verbose_logger.exception, if exception is raised
## Auto-strip advisor blocks from history if advisor tool is absent.
## Prevents Anthropic 400: advisor_tool_result in history requires advisor tool.
_all_tools = optional_params.get("tools") or []
_has_advisor = any(
isinstance(t, dict) and t.get("type") == ANTHROPIC_ADVISOR_TOOL_TYPE
for t in _all_tools
)
if not _has_advisor:
anthropic_messages = strip_advisor_blocks_from_messages(anthropic_messages)
## Add code_execution tool if container_upload is in messages
_tools = (
cast(
@ -1440,23 +1479,32 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
**optional_params,
}
## Handle output_config (Anthropic-specific parameter)
if "output_config" in optional_params:
output_config = optional_params.get("output_config")
if output_config and isinstance(output_config, dict):
effort = output_config.get("effort")
if effort and effort not in ["high", "medium", "low", "max"]:
raise ValueError(
f"Invalid effort value: {effort}. Must be one of: 'high', 'medium', 'low', 'max'"
)
if effort == "max" and not self._is_opus_4_6_model(model):
raise ValueError(
f"effort='max' is only supported by Claude Opus 4.6. Got model: {model}"
)
data["output_config"] = output_config
self._apply_output_config(
data=data, model=model, optional_params=optional_params
)
return data
def _apply_output_config(
self, data: dict, model: str, optional_params: dict
) -> None:
"""Validate and apply output_config to the request data."""
if "output_config" not in optional_params:
return
output_config = optional_params.get("output_config")
if not output_config or not isinstance(output_config, dict):
return
effort = output_config.get("effort")
if effort and effort not in ["high", "medium", "low", "max"]:
raise ValueError(
f"Invalid effort value: {effort}. Must be one of: 'high', 'medium', 'low', 'max'"
)
if effort == "max" and not self._is_opus_4_6_model(model):
raise ValueError(
f"effort='max' is only supported by Claude Opus 4.6. Got model: {model}"
)
data["output_config"] = output_config
def _transform_response_for_json_mode(
self,
json_mode: Optional[bool],

56
litellm/llms/anthropic/common_utils.py
View file

@ -2,7 +2,7 @@
This file contains common utils for anthropic calls.
"""
from typing import Dict, List, Optional, Union
from typing import Any, Dict, List, Optional, Union
import httpx
@ -464,9 +464,9 @@ class AnthropicModelInfo(BaseLLMModelInfo):
if web_search_tool_used:
from litellm.types.llms.anthropic import ANTHROPIC_BETA_HEADER_VALUES
headers[
"anthropic-beta"
] = ANTHROPIC_BETA_HEADER_VALUES.WEB_SEARCH_2025_03_05.value
headers["anthropic-beta"] = (
ANTHROPIC_BETA_HEADER_VALUES.WEB_SEARCH_2025_03_05.value
)
elif len(betas) > 0:
headers["anthropic-beta"] = ",".join(betas)
@ -639,6 +639,54 @@ class AnthropicModelInfo(BaseLLMModelInfo):
return AnthropicTokenCounter()
def strip_advisor_blocks_from_messages(messages: List[Any]) -> List[Any]:
"""
Remove server_tool_use (name='advisor') and advisor_tool_result blocks from
assistant message content when the advisor tool is absent from the request.
Prevents Anthropic 400 invalid_request_error: if advisor_tool_result blocks
exist in history but the advisor tool is not in the tools array, the API rejects
the request. This happens when the user has removed the advisor tool for cost
control or on a follow-up turn.
"""
for message in messages:
if not isinstance(message, dict) or message.get("role") != "assistant":
continue
content = message.get("content")
if not isinstance(content, list):
continue
advisor_ids: set = set()
for block in content:
if (
isinstance(block, dict)
and block.get("type") == "server_tool_use"
and block.get("name") == "advisor"
):
bid = block.get("id")
if bid:
advisor_ids.add(bid)
if not advisor_ids:
continue
message["content"] = [
block
for block in content
if not (
isinstance(block, dict)
and (
(
block.get("type") == "server_tool_use"
and block.get("name") == "advisor"
)
or (
block.get("type") == "advisor_tool_result"
and block.get("tool_use_id") in advisor_ids
)
)
)
]
return messages
def process_anthropic_headers(headers: Union[httpx.Headers, dict]) -> dict:
openai_headers = {}
if "anthropic-ratelimit-requests-limit" in headers:

8
litellm/llms/anthropic/experimental_pass_through/messages/handler.py
View file

@ -187,11 +187,9 @@ async def anthropic_messages(
"""
Async: Make llm api request in Anthropic /messages API spec
"""
# Save original stream flag before pre-request hooks can convert it.
# The websearch interception hook converts stream=True → stream=False
# for the agentic loop, but the short-circuit path needs to know
# whether the caller originally requested streaming.
original_stream = stream
original_stream = stream or kwargs.get(
"_websearch_interception_converted_stream", False
)
# Execute pre-request hooks to allow CustomLoggers to modify request
request_kwargs = await _execute_pre_request_hooks(

32
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
View file

@ -8,6 +8,7 @@ from litellm.llms.base_llm.anthropic_messages.transformation import (
BaseAnthropicMessagesConfig,
)
from litellm.types.llms.anthropic import (
ANTHROPIC_ADVISOR_TOOL_TYPE,
ANTHROPIC_BETA_HEADER_VALUES,
AnthropicMessagesRequest,
)
@ -21,6 +22,7 @@ from ...common_utils import (
AnthropicError,
AnthropicModelInfo,
optionally_handle_anthropic_oauth,
strip_advisor_blocks_from_messages,
)
DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
@ -208,12 +210,23 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
)
)
if transformed_context_management is not None:
anthropic_messages_optional_request_params[
"context_management"
] = transformed_context_management
anthropic_messages_optional_request_params["context_management"] = (
transformed_context_management
)
####### get required params for all anthropic messages requests ######
verbose_logger.debug(f"TRANSFORMATION DEBUG - Messages: {messages}")
# Auto-strip advisor blocks from history if advisor tool is absent.
# Prevents Anthropic 400: advisor_tool_result in history requires advisor tool.
_tools = anthropic_messages_optional_request_params.get("tools") or []
_has_advisor = any(
isinstance(t, dict) and t.get("type") == ANTHROPIC_ADVISOR_TOOL_TYPE
for t in _tools
)
if not _has_advisor:
messages = strip_advisor_blocks_from_messages(messages) # type: ignore[assignment]
anthropic_messages_request: AnthropicMessagesRequest = AnthropicMessagesRequest(
messages=messages,
max_tokens=max_tokens,
@ -324,6 +337,19 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
if optional_params.get("speed") == "fast":
beta_values.add(ANTHROPIC_BETA_HEADER_VALUES.FAST_MODE_2026_02_01.value)
# Check for advisor tool
tools = optional_params.get("tools")
if tools:
for tool in tools:
if (
isinstance(tool, dict)
and tool.get("type") == ANTHROPIC_ADVISOR_TOOL_TYPE
):
beta_values.add(
ANTHROPIC_BETA_HEADER_VALUES.ADVISOR_TOOL_2026_03_01.value
)
break
# Check for tool search tools
tools = optional_params.get("tools")
if tools:

0
litellm/llms/azure/containers/__init__.py Normal file
View file

48
litellm/llms/azure/containers/transformation.py Normal file
View file

@ -0,0 +1,48 @@
from typing import Optional
from litellm.llms.azure.common_utils import BaseAzureLLM
from litellm.llms.openai.containers.transformation import OpenAIContainerConfig
from litellm.types.router import GenericLiteLLMParams
class AzureContainerConfig(OpenAIContainerConfig):
"""
Configuration class for Azure OpenAI container API.
Inherits request/response transformations from OpenAIContainerConfig since
Azure's container API is wire-compatible with OpenAI's. Only overrides
authentication (api-key header) and URL construction (openai/v1/containers path).
Azure container API reference:
https://learn.microsoft.com/en-us/azure/foundry/openai/latest#containers
"""
def validate_environment(
self,
headers: dict,
api_key: Optional[str] = None,
) -> dict:
return BaseAzureLLM._base_validate_azure_environment(
headers=headers,
litellm_params=GenericLiteLLMParams(api_key=api_key),
)
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Build the Azure container endpoint URL.
Azure container API uses the path:
{endpoint}/openai/v1/containers
when api_version is 'v1', 'latest', or 'preview'; otherwise:
{endpoint}/openai/containers
"""
return BaseAzureLLM._get_base_azure_url(
api_base=api_base,
litellm_params=litellm_params,
route="/openai/containers",
default_api_version="v1",
)

24
litellm/llms/base_llm/guardrail_translation/utils.py Normal file
View file

@ -0,0 +1,24 @@
from __future__ import annotations
from typing import Any, List
from litellm.types.llms.openai import AllMessageValues
def effective_skip_system_message_for_guardrail(guardrail_to_apply: Any) -> bool:
per = getattr(guardrail_to_apply, "skip_system_message_in_guardrail", None)
if per is not None:
return bool(per)
import litellm
return bool(getattr(litellm, "skip_system_message_in_guardrail", False))
def openai_messages_without_system(
messages: List[AllMessageValues],
) -> List[AllMessageValues]:
return [
m
for m in messages
if str((m or {}).get("role") or "").lower() != "system"
]

2
litellm/llms/bedrock/base_aws_llm.py
View file

@ -700,7 +700,7 @@ class BaseAWSLLM:
"RoleSessionName": aws_session_name,
"WebIdentityToken": oidc_token,
"DurationSeconds": 3600,
"Policy": '{"Version":"2012-10-17","Statement":[{"Sid":"BedrockLiteLLM","Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":"*","Condition":{"Bool":{"aws:SecureTransport":"true"},"StringLike":{"aws:UserAgent":"litellm/*"}}}]}',
"Policy": '{"Version":"2012-10-17","Statement":[{"Sid":"BedrockLiteLLM","Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream","bedrock:ApplyGuardrail","bedrock:GetGuardrail","bedrock:ListGuardrails"],"Resource":"*","Condition":{"Bool":{"aws:SecureTransport":"true"}}}]}',
}
# Add ExternalId parameter if provided

13
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -538,9 +538,9 @@ class AmazonAnthropicClaudeMessagesConfig(
merges usage from message_start and message_delta but ignores
message_stop. This method buffers message_delta and, when
message_stop arrives with cache usage, merges those fields into the
message_delta usage and also updates the input_tokens on
message_delta to include the full count (uncached + cache_creation +
cache_read).
message_delta usage. input_tokens is kept as the uncached-only
count; downstream calculate_usage adds cache tokens to
prompt_tokens.
"""
_CACHE_FIELDS = ("cache_creation_input_tokens", "cache_read_input_tokens")
pending_delta = None
@ -569,12 +569,7 @@ class AmazonAnthropicClaudeMessagesConfig(
raw_input = stop_usage.get("input_tokens")
if raw_input is not None:
uncached = raw_input if isinstance(raw_input, int) else 0
raw_cc = delta_usage.get("cache_creation_input_tokens", 0)
cache_creation = raw_cc if isinstance(raw_cc, int) else 0
raw_cr = delta_usage.get("cache_read_input_tokens", 0)
cache_read = raw_cr if isinstance(raw_cr, int) else 0
delta_usage["input_tokens"] = uncached + cache_creation + cache_read
delta_usage["input_tokens"] = raw_input if isinstance(raw_input, int) else 0
if delta_usage:
pending_delta["usage"] = delta_usage # type: ignore[arg-type]

21
litellm/llms/custom_httpx/container_handler.py
View file

@ -61,18 +61,29 @@ def _build_url(
) -> str:
"""Build the full URL by substituting path parameters.
The api_base from get_complete_url already includes /containers,
so we need to strip that prefix from the path_template.
The api_base from get_complete_url already includes /containers and may include
query parameters. We need to parse the URL, append the path, then preserve the
query parameters.
"""
# api_base ends with /containers, path_template starts with /containers
# So we need to strip /containers from the path
if path_template.startswith("/containers"):
path_template = path_template[len("/containers") :]
url = f"{api_base.rstrip('/')}{path_template}"
# Substitute path parameters
for param, value in path_params.items():
url = url.replace(f"{{{param}}}", value)
return url
path_template = path_template.replace(f"{{{param}}}", value)
# Parse the api_base to extract existing query params
parsed_base = httpx.URL(api_base)
# Append the path to the existing path (before query params)
new_path = f"{parsed_base.path.rstrip('/')}{path_template}"
# Rebuild URL with new path, preserving query params
final_url = parsed_base.copy_with(path=new_path)
return str(final_url)
def _build_query_params(

11
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -1,5 +1,6 @@
import json
import ssl
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
from typing import (
TYPE_CHECKING,
Any,
@ -5027,6 +5028,16 @@ class BaseLLMHTTPHandler:
litellm_params={},
)
ws_url = http_url.replace("https://", "wss://").replace("http://", "ws://")
# OpenAI's WebSocket responses endpoint requires ?model= in the URL,
# matching the Realtime API convention (wss://.../v1/realtime?model=...).
# Use urllib.parse so existing query params (e.g. api-version) are preserved.
_parsed = urlparse(ws_url)
_qs = parse_qs(_parsed.query)
if "model" not in _qs:
_qs["model"] = [model]
ws_url = urlunparse(
_parsed._replace(query=urlencode({k: v[0] for k, v in _qs.items()}))
)
try:
ssl_context = get_shared_realtime_ssl_context()

1
litellm/llms/gemini/chat/transformation.py
View file

@ -91,6 +91,7 @@ class GoogleAIStudioGeminiConfig(VertexGeminiConfig):
"modalities",
"parallel_tool_calls",
"web_search_options",
"service_tier",
]
if supports_reasoning(model, custom_llm_provider="gemini"):
supported_params.append("reasoning_effort")

15
litellm/llms/litellm_proxy/skills/prompt_injection.py
View file

@ -5,6 +5,7 @@ Handles extraction of skill content (SKILL.md) from stored ZIP files
and injection into the system prompt for non-Anthropic models.
"""
import posixpath
import zipfile
from io import BytesIO
from typing import Any, Dict, List, Optional
@ -103,8 +104,18 @@ class SkillPromptInjectionHandler:
else:
clean_path = name
if clean_path:
files[clean_path] = zf.read(name)
if not clean_path:
continue
# Ensure the path stays within the intended directory
normalized = posixpath.normpath(clean_path)
if normalized.startswith("..") or posixpath.isabs(normalized):
verbose_logger.warning(
f"SkillPromptInjectionHandler: Skipping entry with invalid path in skill {skill.skill_id}: {name}"
)
continue
files[normalized] = zf.read(name)
except Exception as e:
verbose_logger.warning(
f"SkillPromptInjectionHandler: Error extracting files from skill {skill.skill_id}: {e}"

8
litellm/llms/litellm_proxy/skills/sandbox_executor.py
View file

@ -94,9 +94,15 @@ class SkillsSandboxExecutor:
# Create a temp directory to stage files
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir_abs = os.path.abspath(tmpdir)
for path, content in skill_files.items():
# Create the file in temp directory
local_path = os.path.join(tmpdir, path)
local_path = os.path.abspath(os.path.join(tmpdir, path))
if not local_path.startswith(tmpdir_abs + os.sep):
verbose_logger.warning(
f"SkillsSandboxExecutor: Skipping file with invalid path: {path}"
)
continue
os.makedirs(os.path.dirname(local_path), exist_ok=True)
with open(local_path, "wb") as f:
f.write(content)

22
litellm/llms/openai/chat/guardrail_translation/handler.py
View file

@ -19,8 +19,12 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.llms.base_llm.guardrail_translation.utils import (
effective_skip_system_message_for_guardrail,
openai_messages_without_system,
)
from litellm.main import stream_chunk_builder
from litellm.types.llms.openai import ChatCompletionToolParam
from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
from litellm.types.utils import (
Choices,
GenericGuardrailAPIInputs,
@ -57,6 +61,8 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if messages is None:
return data
skip_system = effective_skip_system_message_for_guardrail(guardrail_to_apply)
texts_to_check: List[str] = []
images_to_check: List[str] = []
tool_calls_to_check: List[ChatCompletionToolParam] = []
@ -76,6 +82,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
tool_calls_to_check=tool_calls_to_check,
text_task_mappings=text_task_mappings,
tool_call_task_mappings=tool_call_task_mappings,
skip_system_message=skip_system,
)
# Step 2: Apply guardrail to all texts and tool calls in batch
@ -86,9 +93,12 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if tool_calls_to_check:
inputs["tool_calls"] = tool_calls_to_check # type: ignore
if messages:
inputs[
"structured_messages"
] = messages # pass the openai /chat/completions messages to the guardrail, as-is
msg_list = cast(List[AllMessageValues], messages)
inputs["structured_messages"] = (
openai_messages_without_system(msg_list)
if skip_system
else msg_list
)
# Pass tools (function definitions) to the guardrail
tools = data.get("tools")
if tools:
@ -157,12 +167,16 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
tool_calls_to_check: List[ChatCompletionToolParam],
text_task_mappings: List[Tuple[int, Optional[int]]],
tool_call_task_mappings: List[Tuple[int, int]],
skip_system_message: bool = False,
) -> None:
"""
Extract text content, images, and tool calls from a message.
Override this method to customize text/image/tool call extraction logic.
"""
if skip_system_message and str(message.get("role") or "").lower() == "system":
return
content = message.get("content", None)
if content is not None:
if isinstance(content, str):

11
litellm/llms/openai/containers/transformation.py
View file

@ -17,6 +17,7 @@ from litellm.types.containers.main import (
from litellm.types.router import GenericLiteLLMParams
from ...base_llm.containers.transformation import BaseContainerConfig
from .utils import join_container_api_base_path
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -197,7 +198,7 @@ class OpenAIContainerConfig(BaseContainerConfig):
) -> Tuple[str, Dict]:
"""Transform the OpenAI container retrieve request."""
# For container retrieve, we just need to construct the URL
url = f"{api_base.rstrip('/')}/{container_id}"
url = join_container_api_base_path(api_base, f"/{container_id}")
# No additional data needed for GET request
data: Dict[str, Any] = {}
@ -229,7 +230,7 @@ class OpenAIContainerConfig(BaseContainerConfig):
- DELETE /v1/containers/{container_id}
"""
# Construct the URL for container delete
url = f"{api_base.rstrip('/')}/{container_id}"
url = join_container_api_base_path(api_base, f"/{container_id}")
# No data needed for DELETE request
data: Dict[str, Any] = {}
@ -266,7 +267,7 @@ class OpenAIContainerConfig(BaseContainerConfig):
- GET /v1/containers/{container_id}/files
"""
# Construct the URL for container files
url = f"{api_base.rstrip('/')}/{container_id}/files"
url = join_container_api_base_path(api_base, f"/{container_id}/files")
# Prepare query parameters
params: Dict[str, Any] = {}
@ -310,7 +311,9 @@ class OpenAIContainerConfig(BaseContainerConfig):
- GET /v1/containers/{container_id}/files/{file_id}/content
"""
# Construct the URL for container file content
url = f"{api_base.rstrip('/')}/{container_id}/files/{file_id}/content"
url = join_container_api_base_path(
api_base, f"/{container_id}/files/{file_id}/content"
)
# No query parameters needed
params: Dict[str, Any] = {}

18
litellm/llms/openai/containers/utils.py Normal file
View file

@ -0,0 +1,18 @@
"""Shared helpers for OpenAI-compatible container API URL construction."""
import httpx
def join_container_api_base_path(api_base: str, path_suffix: str) -> str:
"""Append ``path_suffix`` to the path of ``api_base``, keeping the query string last.
Azure (and some bases) pass ``api_base`` like
``https://host/openai/v1/containers?api-version=v1``. Naive string concat would
produce ``...?api-version=v1/cntr_...`` which is invalid; this uses ``httpx.URL``
so the result is ``.../containers/cntr_.../files?api-version=v1``.
"""
if not path_suffix.startswith("/"):
path_suffix = f"/{path_suffix}"
parsed = httpx.URL(api_base)
new_path = f"{parsed.path.rstrip('/')}{path_suffix}"
return str(parsed.copy_with(path=new_path))

89
litellm/llms/openai/openai.py
View file

@ -32,6 +32,7 @@ import litellm
from litellm import LlmProviders
from litellm._logging import verbose_logger
from litellm.constants import DEFAULT_MAX_RETRIES
from litellm.files.types import FileContentStreamingResult
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
@ -1751,6 +1752,92 @@ class OpenAIFilesAPI(BaseLLM):
return HttpxBinaryResponseContent(response=response.response)
async def afile_content_streaming(
self,
file_content_request: FileContentRequest,
openai_client: AsyncOpenAI,
chunk_size: int = 1024 * 1024,
) -> FileContentStreamingResult:
response_cm = openai_client.files.with_streaming_response.content(
**file_content_request
)
response = await response_cm.__aenter__()
headers = dict(response.headers)
async def _stream() -> AsyncIterator[bytes]:
exc: Optional[BaseException] = None
try:
async for chunk in response.iter_bytes(chunk_size=chunk_size):
yield chunk
except BaseException as e:
exc = e
raise
finally:
if exc is None:
await response_cm.__aexit__(None, None, None)
else:
await response_cm.__aexit__(type(exc), exc, exc.__traceback__)
return FileContentStreamingResult(stream_iterator=_stream(), headers=headers)
def file_content_streaming(
self,
_is_async: bool,
file_content_request: FileContentRequest,
api_base: str,
api_key: Optional[str],
timeout: Union[float, httpx.Timeout],
max_retries: Optional[int],
organization: Optional[str],
chunk_size: int = 1024 * 1024,
client: Optional[Union[OpenAI, AsyncOpenAI]] = None,
) -> FileContentStreamingResult:
openai_client: Optional[Union[OpenAI, AsyncOpenAI]] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
timeout=timeout,
max_retries=max_retries,
organization=organization,
client=client,
_is_async=_is_async,
)
if openai_client is None:
raise ValueError(
"OpenAI client is not initialized. Make sure api_key is passed or OPENAI_API_KEY is set in the environment."
)
if _is_async is True:
if not isinstance(openai_client, AsyncOpenAI):
raise ValueError(
"OpenAI client is not an instance of AsyncOpenAI. Make sure you passed an AsyncOpenAI client."
)
return self.afile_content_streaming( # type: ignore
file_content_request=file_content_request,
openai_client=openai_client,
chunk_size=chunk_size,
)
response_cm = cast(OpenAI, openai_client).files.with_streaming_response.content(
**file_content_request
)
response = response_cm.__enter__()
headers = dict(response.headers)
def _stream() -> Iterator[bytes]:
exc: Optional[BaseException] = None
try:
yield from response.iter_bytes(chunk_size=chunk_size)
except BaseException as e:
exc = e
raise
finally:
if exc is None:
response_cm.__exit__(None, None, None)
else:
response_cm.__exit__(type(exc), exc, exc.__traceback__)
return FileContentStreamingResult(stream_iterator=_stream(), headers=headers)
async def aretrieve_file(
self,
file_id: str,
@ -3045,4 +3132,4 @@ class OpenAIAssistantsAPI(BaseLLM):
tools=tools,
)
return response
return response

0
litellm/llms/sap/__init__.py Normal file
View file

623
litellm/llms/sap/chat/models.py
View file

@ -1,6 +1,8 @@
from typing import Union, Literal
from typing import Union, Literal, Optional
from enum import Enum
import warnings
from pydantic import BaseModel, Field, field_validator
from pydantic import BaseModel, Field, field_validator, model_validator
def validate_different_content(v: Union[str, dict, list]) -> str:
@ -20,7 +22,7 @@ def validate_different_content(v: Union[str, dict, list]) -> str:
elif isinstance(v, str):
return v
raise ValueError("Content must be a string")
return v
class TextContent(BaseModel):
@ -49,6 +51,10 @@ class FunctionTool(BaseModel):
parameters: dict = {"type": "object", "properties": {}}
strict: bool = False
def model_dump(self, **kwargs) -> dict:
kwargs["exclude_unset"] = False
return super().model_dump(**kwargs)
@field_validator("parameters", mode="before")
@classmethod
def ensure_object_type(cls, v: dict) -> dict:
@ -66,6 +72,10 @@ class ChatCompletionTool(BaseModel):
type_: Literal["function"] = Field(default="function", alias="type")
function: FunctionTool
def model_dump(self, **kwargs) -> dict:
kwargs["exclude_unset"] = False
return super().model_dump(**kwargs)
class MessageToolCall(BaseModel):
id: str
@ -114,6 +124,9 @@ class SAPToolChatMessage(BaseModel):
)
ChatMessage = Union[SAPMessage, SAPUserMessage, SAPAssistantMessage, SAPToolChatMessage]
class ResponseFormat(BaseModel):
type_: Literal["text", "json_object"] = Field(default="text", alias="type")
@ -128,3 +141,607 @@ class JSONResponseSchema(BaseModel):
class ResponseFormatJSONSchema(BaseModel):
type_: Literal["json_schema"] = Field(default="json_schema", alias="type")
json_schema: JSONResponseSchema
class KeyValueListPair(BaseModel):
key: str
value: list[str]
class DocumentMetadataKeyValueListPairs(KeyValueListPair):
select_mode: Optional[list[Literal["ignoreIfKeyAbsent"]]] = None
class GroundingSearchConfig(BaseModel):
max_chunk_count: Optional[int] = Field(default=None, ge=0)
max_document_count: Optional[int] = Field(default=None, ge=0)
@model_validator(mode="after")
def validate_max_chunk_count_and_max_document_count(self):
if self.max_chunk_count is not None and self.max_document_count is not None:
raise ValueError("Cannot specify both maxChunkCount and maxDocumentCount.")
return self
class DocumentGroundingFilter(BaseModel):
id_: Optional[str] = Field(default=None, alias="id")
data_repository_type: Literal["vector", "help.sap.com"]
search_config: Optional[GroundingSearchConfig] = None
data_repositories: Optional[list[str]] = None
data_repository_metadata: Optional[list[KeyValueListPair]] = None
document_metadata: Optional[list[DocumentMetadataKeyValueListPairs]] = None
chunk_metadata: Optional[list[KeyValueListPair]] = None
class DocumentGroundingPlaceholders(BaseModel):
input: list[str] = Field(min_length=1)
output: str
class DocumentGroundingConfig(BaseModel):
filters: Optional[list[DocumentGroundingFilter]] = None
placeholders: DocumentGroundingPlaceholders
metadata_params: Optional[list[str]] = None
class GroundingModuleConfig(BaseModel):
type_: Literal["document_grounding_service"] = Field(
default="document_grounding_service", alias="type"
)
config: DocumentGroundingConfig
class Template(BaseModel):
template: list[ChatMessage]
defaults: Optional[dict[str, str]] = None
response_format: Optional[Union[ResponseFormat, ResponseFormatJSONSchema]] = None
tools: Optional[list[ChatCompletionTool]] = None
class LLMModelDetails(BaseModel):
name: str
version: str = "latest"
params: Optional[dict] = None
class PromptTemplatingModuleConfig(BaseModel):
prompt: Template
model: LLMModelDetails
class SAPMaskingProfileEntity(str, Enum):
"""
Enumerates the entity categories that can be masked by the SAP Data Privacy Integration service.
This enum lists different types of personal or sensitive information (PII) that can be detected and masked
by the data masking module, such as personal details, organizational data, contact information, and identifiers.
Values:
PERSON: Represents personal names.
ORG: Represents organizational names.
UNIVERSITY: Represents educational institutions.
LOCATION: Represents geographical locations.
EMAIL: Represents email addresses.
PHONE: Represents phone numbers.
ADDRESS: Represents physical addresses.
SAP_IDS_INTERNAL: Represents internal SAP identifiers.
SAP_IDS_PUBLIC: Represents public SAP identifiers.
URL: Represents URLs.
USERNAME_PASSWORD: Represents usernames and passwords.
NATIONAL_ID: Represents national identification numbers.
IBAN: Represents International Bank Account Numbers.
SSN: Represents Social Security Numbers.
CREDIT_CARD_NUMBER: Represents credit card numbers.
PASSPORT: Represents passport numbers.
DRIVING_LICENSE: Represents driving license numbers.
NATIONALITY: Represents nationality information.
RELIGIOUS_GROUP: Represents religious group affiliation.
POLITICAL_GROUP: Represents political group affiliation.
PRONOUNS_GENDER: Represents pronouns and gender identity.
GENDER: Represents gender information.
SEXUAL_ORIENTATION: Represents sexual orientation.
TRADE_UNION: Represents trade union membership.
SENSITIVE_DATA: Represents any other sensitive information.
"""
PERSON = "profile-person"
ORG = "profile-org"
UNIVERSITY = "profile-university"
LOCATION = "profile-location"
EMAIL = "profile-email"
PHONE = "profile-phone"
ADDRESS = "profile-address"
SAP_IDS_INTERNAL = "profile-sapids-internal"
SAP_IDS_PUBLIC = "profile-sapids-public"
URL = "profile-url"
USERNAME_PASSWORD = "profile-username-password"
NATIONAL_ID = "profile-nationalid"
IBAN = "profile-iban"
SSN = "profile-ssn"
CREDIT_CARD_NUMBER = "profile-credit-card-number"
PASSPORT = "profile-passport"
DRIVING_LICENSE = "profile-driverlicense"
NATIONALITY = "profile-nationality"
RELIGIOUS_GROUP = "profile-religious-group"
POLITICAL_GROUP = "profile-political-group"
PRONOUNS_GENDER = "profile-pronouns-gender"
GENDER = "profile-gender"
SEXUAL_ORIENTATION = "profile-sexual-orientation"
TRADE_UNION = "profile-trade-union"
SENSITIVE_DATA = "profile-sensitive-data"
ETHNICITY = "profile-ethnicity"
class DPIMethodConstant(BaseModel):
"""
Replaces the entity with the specified value followed by an incrementing number
"""
method: Literal["constant"] = "constant"
value: str
class DPIMethodFabricatedData(BaseModel):
"""
Replaces the entity with a randomly generated value appropriate to its type.
"""
method: Literal["fabricated_data"] = "fabricated_data"
class DPICustomEntity(BaseModel):
"""
regex: Regular expression to match the entity
replacement_strategy: Replacement strategy to be used for the entity
"""
regex: str
replacement_strategy: DPIMethodConstant
class DPIStandardEntity(BaseModel):
"""
type: Standard entity type to be masked
replacement_strategy: Replacement strategy to be used for the entity
"""
type_: SAPMaskingProfileEntity = Field(..., alias="type")
replacement_strategy: Optional[
Union[DPIMethodConstant, DPIMethodFabricatedData]
] = None
class MaskGroundingInput(BaseModel):
"""
Controls whether the input to the grounding module will be masked with the configuration
supplied in the masking module
"""
enabled: bool = False
class MaskingProviderConfig(BaseModel):
"""
SAP Data Privacy Integration provider for data masking.
This class implements the SAP Data Privacy Integration service, which can anonymize or pseudonymize
specified entity categories in the input data. It supports masking sensitive information like personal names,
contact details, and identifiers.
Args:
method: The method of masking to apply (anonymization or pseudonymization).
entities: A list of entity categories to be masked, such as names, locations, or emails.
allowlist: A list of strings that should not be masked.
mask_grounding_input: A flag indicating whether to mask input to the grounding module.
"""
type_: Literal["sap_data_privacy_integration"] = Field(
default="sap_data_privacy_integration", alias="type"
)
method: Literal["anonymization", "pseudonymization"]
entities: list[Union[DPIStandardEntity, DPICustomEntity]]
allowlist: Optional[list[str]] = None
mask_grounding_input: Optional[MaskGroundingInput] = None
class MaskingModuleConfig(BaseModel):
"""
Configuration for the data masking module.
Args:
providers: list of masking service provider configurations
masking_providers: list of masking provider configurations
IMPORTANT: use exactly one of the parameters to set the list of masking provider configurations.
DEPRECATED: parameter 'masking_providers' will be removed Sept 15, 2026. Use 'providers' instead.
"""
providers: Optional[list[MaskingProviderConfig]] = Field(min_length=1, default=None)
masking_providers: Optional[list[MaskingProviderConfig]] = Field(
min_length=1, default=None
)
@model_validator(mode="after")
def enforce_exactly_one_provider_list(self):
has_providers = self.providers is not None
has_masking_providers = self.masking_providers is not None
if not has_providers and not has_masking_providers:
raise ValueError(
"For SAP Masking Module Config you must provide 'providers'."
)
if has_providers and has_masking_providers:
raise ValueError(
"For SAP Masking Module Config you must set exactly one of: 'providers' or 'masking_providers', not both."
)
if has_masking_providers:
warnings.warn(
"The 'masking_providers' parameter is deprecated and will be removed on Sept 15, 2026. "
"Use 'providers' instead.",
DeprecationWarning,
stacklevel=5,
)
return self
class AzureThreshold(int, Enum):
"""
Enumerates the threshold levels for the Azure Content Safety service.
This enum defines the various threshold levels that can be used to filter
content based on its safety score. Each threshold value represents a specific
level of content moderation.
Values:
ALLOW_SAFE: Allows only Safe content.
ALLOW_SAFE_LOW: Allows Safe and Low content.
ALLOW_SAFE_LOW_MEDIUM: Allows Safe, Low, and Medium content.
ALLOW_ALL: Allows all content (Safe, Low, Medium, and High).
"""
ALLOW_SAFE = 0
ALLOW_SAFE_LOW = 2
ALLOW_SAFE_LOW_MEDIUM = 4
ALLOW_ALL = 6
class AzureContentFilter(BaseModel):
"""
Specific filter configuration for Azure Content Safety.
This class configures content filtering based on Azure's categories and
severity levels. It allows setting thresholds for hate speech, sexual content,
violence, and self-harm content.
Values:
hate: Threshold for hate speech content.
sexual: Threshold for sexual content.
violence: Threshold for violent content.
self_harm: Threshold for self-harm content.
"""
hate: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
sexual: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
violence: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
self_harm: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
class AzureContentSafetyInput(AzureContentFilter):
"""
Filter configuration for Azure Content Safety Input
Values:
hate: Threshold for hate speech content.
sexual: Threshold for sexual content.
violence: Threshold for violent content.
self_harm: Threshold for self-harm content.
prompt_shield: A flag to use prompt shield
"""
prompt_shield: Optional[bool] = False
class AzureContentSafetyOutput(AzureContentFilter):
"""
Filter configuration for Azure Content Safety Output
Values:
hate: Threshold for hate speech content.
sexual: Threshold for sexual content.
violence: Threshold for violent content.
self_harm: Threshold for self-harm content.
protected_material_code: Detect protected code content from known GitHub repositories.
The scan includes software libraries, source code, algorithms,
and other proprietary programming content.
"""
protected_material_code: Optional[bool] = False
class LlamaGuard38bFilter(BaseModel):
"""
Specific implementation of ContentFilter for Llama Guard 3. Llama Guard 3 is a
Llama-3.1-8B pretrained model, fine-tuned for content safety classification.
Args:
violent_crimes: Responses that enable, encourage, or endorse the commission of violent crimes.
non_violent_crimes: Responses that enable, encourage, or endorse the commission of non-violent crimes.
sex_crimes: Responses that enable, encourage, or endorse the commission of sex-related crimes.
child_exploitation: Responses that contain, describe, enable or endorse the sexual abuse of children.
defamation: Responses that are both verifiably false and likely to injure a living person's reputation.
specialized_advice: Responses that contain specialized financial, medical or legal advice.
privacy: Responses that contain sensitive or nonpublic personal information.
intellectual_property: Responses that may violate the intellectual property rights of any third party.
indiscriminate_weapons: Responses that enable, encourage, or endorse the creation of indiscriminate weapons.
hate: Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics.
self_harm: Responses that enable, encourage, or endorse acts of intentional self-harm.
sexual_content: Responses that contain erotica.
elections: Responses that contain factually incorrect information about electoral systems and processes.
code_interpreter_abuse: Responses that seek to abuse code interpreters.
"""
violent_crimes: bool = Field(default=False)
non_violent_crimes: bool = Field(default=False)
sex_crimes: bool = Field(default=False)
child_exploitation: bool = Field(default=False)
defamation: bool = Field(default=False)
specialized_advice: bool = Field(default=False)
privacy: bool = Field(default=False)
intellectual_property: bool = Field(default=False)
indiscriminate_weapons: bool = Field(default=False)
hate: bool = Field(default=False)
self_harm: bool = Field(default=False)
sexual_content: bool = Field(default=False)
elections: bool = Field(default=False)
code_interpreter_abuse: bool = Field(default=False)
class LlamaGuard38bFilterConfig(BaseModel):
type_: Literal["llama_guard_3_8b"] = Field(default="llama_guard_3_8b", alias="type")
config: LlamaGuard38bFilter
class AzureContentSafetyInputFilterConfig(BaseModel):
type_: Literal["azure_content_safety"] = Field(
default="azure_content_safety", alias="type"
)
config: Optional[AzureContentSafetyInput] = None
class AzureContentSafetyOutputFilterConfig(BaseModel):
type_: Literal["azure_content_safety"] = Field(
default="azure_content_safety", alias="type"
)
config: Optional[AzureContentSafetyOutput] = None
class FilteringStreamOptions(BaseModel):
"""
overlap: Number of characters that should be additionally sent to content filtering services
from previous chunks as additional context.
"""
overlap: Optional[int] = Field(default=0, ge=0, le=10000)
class InputFiltering(BaseModel):
"""Module for managing and applying input content filters.
Args:
filters: List of ContentFilter objects to be applied to input content.
"""
filters: list[
Union[AzureContentSafetyInputFilterConfig, LlamaGuard38bFilterConfig]
] = Field(min_length=1)
class OutputFiltering(BaseModel):
"""Module for managing and applying output content filters.
Args:
filters: List of ContentFilter objects to be applied to output content.
stream_options: Module-specific streaming options.
"""
filters: list[
Union[AzureContentSafetyOutputFilterConfig, LlamaGuard38bFilterConfig]
] = Field(min_length=1)
stream_options: Optional[FilteringStreamOptions] = None
class FilteringModuleConfig(BaseModel):
"""Module for managing and applying content filters.
Args:
input: Module for filtering and validating input content before processing.
output: Module for filtering and validating output content after generation.
"""
input: Optional[InputFiltering] = None
output: Optional[OutputFiltering] = None
@model_validator(mode="after")
def enforce_min_properties(self) -> "FilteringModuleConfig":
"""
Ensure at least one of input or output filtering is provided.
"""
if self.input is None and self.output is None:
raise ValueError(
"For using SAP Filtering Module you must provide at least one property: input or output filters."
)
return self
class SAPDocumentTranslationApplyToSelector(BaseModel):
"""
This selector allows you to define the scope of translation, such as specific placeholders or
messages with specific roles.
For example, {"category": "placeholders",
"items": ["user_input"],
"source_language": "de-DE"}
targets the value of "user_input" in placeholder_values specified in the request payload;
and considers the value to be in German.
"""
category: Literal["placeholders", "template_roles"]
items: list[str]
source_language: str
class InputTranslationConfig(BaseModel):
"""
Configuration for input translation.
Args:
source_language: Language of the text to be translated. Example: de-DE
target_language: Language to which the text should be translated. Example: en-US
apply_to: List of selectors that define the scope of translation.
"""
source_language: Optional[str] = None
target_language: str
apply_to: Optional[list[SAPDocumentTranslationApplyToSelector]] = None
class OutputTranslationConfig(BaseModel):
source_language: Optional[str] = None
target_language: Union[str, SAPDocumentTranslationApplyToSelector]
class SAPDocumentTranslationInput(BaseModel):
"""
Configuration for input translation
Args:
type: The type of translation module (e.g., 'sap_document_translation').
translate_messages_history: If true, the messages history will be translated as well.
config: Configuration object for the translation module.
"""
type_: Literal["sap_document_translation"] = Field(
default="sap_document_translation", alias="type"
)
translate_messages_history: Optional[bool] = None
config: InputTranslationConfig
class SAPDocumentTranslationOutput(BaseModel):
"""
Configuration for output translation
Args:
type: The type of translation module (e.g., 'sap_document_translation').
config: Configuration object for the translation module.
"""
type_: Literal["sap_document_translation"] = Field(
default="sap_document_translation", alias="type"
)
config: OutputTranslationConfig
class TranslationModuleConfig(BaseModel):
"""
Configuration for translation module
Args:
input: Configuration for input translation
output: Configuration for output translation
"""
input: Optional[SAPDocumentTranslationInput] = None
output: Optional[SAPDocumentTranslationOutput] = None
@model_validator(mode="after")
def enforce_min_properties(self) -> "TranslationModuleConfig":
if self.input is None and self.output is None:
raise ValueError(
"TranslationModuleConfig requires at least one of 'input' or 'output'."
)
return self
class ModuleConfig(BaseModel):
prompt_templating: PromptTemplatingModuleConfig
filtering: Optional[FilteringModuleConfig] = None
masking: Optional[MaskingModuleConfig] = None
grounding: Optional[GroundingModuleConfig] = None
translation: Optional[TranslationModuleConfig] = None
class GlobalStreamOptions(BaseModel):
enabled: bool = False
chunk_size: Optional[int] = Field(default=None, ge=1)
delimiters: Optional[list[str]] = None
class OrchestrationConfig(BaseModel):
modules: Union[ModuleConfig, list[ModuleConfig]]
stream: Optional[GlobalStreamOptions] = None
class OrchestrationRequest(BaseModel):
config: OrchestrationConfig
placeholder_values: Optional[dict[str, str]] = None

230
litellm/llms/sap/chat/transformation.py
View file

@ -11,6 +11,7 @@ from typing import (
TYPE_CHECKING,
Iterator,
AsyncIterator,
FrozenSet,
)
from functools import cached_property
import litellm
@ -31,12 +32,13 @@ else:
from ..credentials import get_token_creator
from .models import (
SAPMessage,
SAPAssistantMessage,
SAPToolChatMessage,
ChatCompletionTool,
ResponseFormatJSONSchema,
OrchestrationRequest,
ResponseFormat,
ResponseFormatJSONSchema,
SAPAssistantMessage,
SAPMessage,
SAPToolChatMessage,
SAPUserMessage,
)
from .handler import (
@ -45,9 +47,65 @@ from .handler import (
SAPStreamIterator,
)
# Keys routed outside SAP orchestration `model.params` (prompt, stream, fallbacks, etc.)
_SAP_MODEL_PARAMS_EXCLUDED_KEYS: FrozenSet[str] = frozenset(
{
"tools",
"tool_choice",
"stream_options",
"fallback_sap_modules",
"placeholder_values",
"model_version",
}
)
def validate_dict(data: dict, model) -> dict:
return model(**data).model_dump(by_alias=True)
return model(**data).model_dump(by_alias=True, exclude_unset=True)
def _messages_to_sap_template(messages: List[Dict[str, str]]) -> list: # type: ignore[type-arg]
template = []
for message in messages:
if message["role"] == "user":
template.append(validate_dict(message, SAPUserMessage))
elif message["role"] == "assistant":
template.append(validate_dict(message, SAPAssistantMessage))
elif message["role"] == "tool":
template.append(validate_dict(message, SAPToolChatMessage))
else:
template.append(validate_dict(message, SAPMessage))
return template
def _tools_response_format_and_stream(
optional_params: dict, model_params: dict
) -> Tuple[dict, dict, dict]:
tools_ = optional_params.pop("tools", [])
tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_]
tools: dict = {"tools": tools_} if tools_ else {}
response_format = model_params.pop("response_format", {})
resp_type = response_format.get("type", None)
if resp_type:
if resp_type == "json_schema":
response_format = validate_dict(
response_format, ResponseFormatJSONSchema
)
else:
response_format = validate_dict(response_format, ResponseFormat)
response_format = {"response_format": response_format}
model_params.pop("stream", False)
stream_config: dict = {}
if "stream_options" in optional_params:
stream_options = optional_params.pop("stream_options", {})
if "chunk_size" in stream_options:
stream_config["chunk_size"] = stream_options.get("chunk_size")
if "delimiters" in stream_options:
stream_config["delimiters"] = stream_options.get("delimiters")
return tools, response_format, stream_config
class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
@ -208,48 +266,25 @@ class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
api_base_ = f"{self.deployment_url}/v2/completion"
return api_base_
def transform_request(
def _build_prompt_module(
self,
model: str,
messages: List[Dict[str, str]], # type: ignore
optional_params: dict,
litellm_params: dict,
headers: dict,
model_name: str,
template_messages: List[Dict[str, str]],
params: dict,
) -> dict:
# Filter out parameters that are not valid model params for SAP Orchestration API
# - tools, model_version, deployment_url: handled separately
excluded_params = {"tools", "model_version", "deployment_url"}
# Filter strict for GPT models only - SAP AI Core doesn't accept it as a model param
# LangChain agents pass strict=true at top level, which fails for GPT models
# Anthropic models accept strict, so preserve it for them
if model.startswith("gpt"):
excluded_params.add("strict")
if model_name.startswith("gpt") and "strict" in params:
params.pop("strict")
model_params = {
k: v for k, v in optional_params.items() if k not in excluded_params
}
model_version = params.pop("model_version", "latest")
model_version = optional_params.pop("model_version", "latest")
template = []
for message in messages:
if message["role"] == "user":
template.append(validate_dict(message, SAPUserMessage))
elif message["role"] == "assistant":
template.append(validate_dict(message, SAPAssistantMessage))
elif message["role"] == "tool":
template.append(validate_dict(message, SAPToolChatMessage))
else:
template.append(validate_dict(message, SAPMessage))
tools_ = optional_params.pop("tools", [])
tools_ = params.pop("tools", [])
tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_]
if tools_ != []:
tools = {"tools": tools_}
else:
tools = {}
tools = {"tools": tools_} if tools_ else {}
response_format = model_params.pop("response_format", {})
response_format = params.pop("response_format", {})
resp_type = response_format.get("type", None)
if resp_type:
if resp_type == "json_schema":
@ -259,33 +294,104 @@ class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
else:
response_format = validate_dict(response_format, ResponseFormat)
response_format = {"response_format": response_format}
model_params.pop("stream", False)
stream_config = {}
if "stream_options" in model_params:
# stream_config["enabled"] = True
stream_options = model_params.pop("stream_options", {})
stream_config["chunk_size"] = stream_options.get("chunk_size", 100)
if "delimiters" in stream_options:
stream_config["delimiters"] = stream_options.get("delimiters")
# else:
# stream_config["enabled"] = False
config = {
"config": {
"modules": {
"prompt_templating": {
"prompt": {"template": template, **tools, **response_format},
"model": {
"name": model,
"params": model_params,
"version": model_version,
},
},
else:
response_format = {}
placeholder_defaults = params.pop("placeholder_defaults", {})
placeholder_defaults = (
{"defaults": placeholder_defaults} if placeholder_defaults else {}
)
optional_modules = {}
optional_modules_lst = ["grounding", "masking", "filtering", "translation"]
for module in optional_modules_lst:
if params.get(module, None) is not None:
optional_modules[module] = params.pop(module)
return {
"prompt_templating": {
"prompt": {
"template": template_messages,
**placeholder_defaults,
**tools,
**response_format,
},
"stream": stream_config,
}
"model": {
"name": model_name,
"params": params,
"version": model_version,
},
},
**optional_modules,
}
return config
def transform_request(
self,
model: str,
messages: List[Dict[str, str]], # type: ignore
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
optional_params = dict(optional_params)
optional_params.pop("deployment_url", None)
template = _messages_to_sap_template(messages)
placeholder_values = optional_params.pop("placeholder_values", None)
fallback_modules = optional_params.pop("fallback_sap_modules", [])
optional_params.pop("stream", None)
stream_config: dict = {}
if "stream_options" in optional_params:
stream_options = optional_params.pop("stream_options", {})
if "chunk_size" in stream_options:
stream_config["chunk_size"] = stream_options["chunk_size"]
if "delimiters" in stream_options:
stream_config["delimiters"] = stream_options["delimiters"]
optional_params.pop("tool_choice", None)
modules = [
self._build_prompt_module(
model_name=model,
template_messages=template,
params=dict(optional_params),
)
]
for modules_dict in fallback_modules:
modules_dict = dict(modules_dict)
fallback_model = modules_dict.pop("model", None)
if fallback_model is None:
raise ValueError(
"Each entry in `fallback_sap_modules` must include a 'model' key."
)
if fallback_model.startswith("sap/"):
fallback_model = fallback_model[4:]
fallback_template = modules_dict.pop("messages", [])
modules.append(
self._build_prompt_module(
model_name=fallback_model,
template_messages=fallback_template,
params=modules_dict,
)
)
config_payload: Dict[str, Any] = {
"modules": modules if len(modules) > 1 else modules[0],
}
if stream_config:
config_payload["stream"] = stream_config
request_body: Dict[str, Any] = {"config": config_payload}
if placeholder_values is not None:
request_body["placeholder_values"] = placeholder_values
body = validate_dict(request_body, OrchestrationRequest)
return body
def transform_response(
self,

379
litellm/llms/sap/credentials.py
View file

@ -1,5 +1,5 @@
from __future__ import annotations
from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple
from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple, Union
from datetime import datetime, timedelta, timezone
from threading import Lock
from pathlib import Path
@ -7,9 +7,11 @@ from dataclasses import dataclass
import json
import os
import tempfile
import httpx
from litellm import sap_service_key
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
from litellm.llms.custom_httpx.http_handler import _get_httpx_client, HTTPHandler
from litellm._logging import verbose_logger
import litellm
AUTH_ENDPOINT_SUFFIX = "/oauth/token"
@ -28,11 +30,25 @@ def _get_home() -> str:
return os.getenv(HOME_PATH_ENV_VAR, DEFAULT_HOME_PATH)
def _get_nested(d: Dict[str, Any], path: Sequence[str]) -> Any:
def _get_nested(d: Union[Dict[str, Any], str], path: Sequence[str]) -> Any:
cur: Any = d
if isinstance(cur, str):
# This shouldn't happen if service keys are pre-parsed correctly
try:
cur = json.loads(cur)
except json.JSONDecodeError:
verbose_logger.warning(
"SAP service key or VCAP service is a string but not valid JSON."
)
return None
for k in path:
if not isinstance(cur, dict) or k not in cur:
raise KeyError(".".join(path))
if not isinstance(cur, dict):
verbose_logger.warning(
f"SAP service key or VCAP service traversal hit non-dict type '{type(cur).__name__}' at key '{k}'."
)
return None
if k not in cur:
return None
cur = cur[k]
return cur
@ -47,6 +63,13 @@ def _load_json_env(var_name: str) -> Optional[Dict[str, Any]]:
return None
def _str_or_none(value) -> Optional[str]:
try:
return str(value) if value is not None else None
except Exception:
return None
def _load_vcap() -> Dict[str, Any]:
return _load_json_env(VCAP_SERVICES_ENV_VAR) or {}
@ -59,6 +82,12 @@ def _get_vcap_service(label: str) -> Optional[Dict[str, Any]]:
return None
@dataclass
class Source:
name: str
get: Callable[[CredentialsValue], Optional[str]]
@dataclass(frozen=True)
class CredentialsValue:
name: str
@ -82,7 +111,6 @@ CREDENTIAL_VALUES: Final[List[CredentialsValue]] = [
transform_fn=lambda url: url.rstrip("/")
+ ("" if url.endswith("/v2") else "/v2"),
),
CredentialsValue("resource_group", default="default"),
CredentialsValue(
"cert_url",
("certurl",),
@ -145,81 +173,239 @@ def _env_name(name: str) -> str:
return f"AICORE_{name.upper()}"
def _resolve_value(
cred: CredentialsValue,
*,
kwargs: Dict[str, Any],
env: Dict[str, str],
config: Dict[str, Any],
service_like: Optional[Dict[str, Any]],
) -> Optional[str]:
# 1) explicit kwargs
if cred.name in kwargs and kwargs[cred.name] is not None:
return kwargs[cred.name]
def extract_credentials(source: Source) -> Dict[str, str]:
"""Extract all credentials from a source."""
credentials = {}
for cv in CREDENTIAL_VALUES:
value = source.get(cv)
if value is not None:
credentials[cv.name] = cv.transform_fn(value) if cv.transform_fn else value
return credentials
# 2) environment variables (primary name)
env_key = _env_name(cred.name)
if env_key in env and env[env_key] is not None:
return env[env_key]
# 3) config file (accept both prefixed and plain keys)
for key in (env_key, cred.name):
if key in config and config[key] is not None:
return config[key]
def resolve_credentials(sources: List[Source]) -> Dict[str, str]:
"""Extract credentials from the first source that has any defined."""
for source in sources:
credentials = extract_credentials(source)
if credentials:
verbose_logger.debug(f"Resolved SAP credentials from source {source.name}")
return credentials
raise ValueError("No credentials found in any source")
# 4) service-like source (AICORE_SERVICE_KEY first, else VCAP)
if service_like and cred.vcap_key:
def resolve_resource_group(sources: List[Source]) -> Optional[str]:
"""Find resource_group from the first source that defines it."""
rg_cred = CredentialsValue("resource_group", default="default")
for source in sources:
value = source.get(rg_cred)
if value is not None:
verbose_logger.debug(
f"Resolved GEN AI Hub resource_group from source {source.name}"
)
return value
return rg_cred.default
def _parse_service_key_once(
service_key: Optional[Union[str, dict]]
) -> Optional[Dict[str, Any]]:
"""
Pre-parse service_key if it's a string to avoid repeated JSON parsing.
Returns None if parsing fails (other credential sources may still work).
"""
if service_key is None:
return None
if isinstance(service_key, dict):
return service_key
if isinstance(service_key, str):
try:
val = _get_nested(service_like, ("credentials",) + cred.vcap_key)
if val is not None:
return val
except KeyError:
pass
return json.loads(service_key)
except json.JSONDecodeError:
verbose_logger.warning(
"SAP service key is a string but not valid JSON. Skipping this source."
)
return None
verbose_logger.warning(
f"SAP service key has unexpected type '{type(service_key).__name__}'. Expected str or dict. Ignoring."
)
return None
# 5) default
return cred.default
def _resolve_credential_from_service_key(
service_key: Optional[Union[str, dict]], cv: CredentialsValue
) -> Optional[str]:
if service_key is None:
return None
val = _str_or_none(
_get_nested(
service_key, (("credentials",) + cv.vcap_key) if cv.vcap_key else (cv.name,)
)
)
if val is None:
return _str_or_none(
_get_nested(service_key, cv.vcap_key if cv.vcap_key else (cv.name,))
)
return val
def fetch_credentials(
service_key: Optional[str] = None, profile: Optional[str] = None, **kwargs
service_key: Optional[Union[str, dict]] = None,
profile: Optional[str] = None,
**kwargs,
) -> Dict[str, str]:
"""
Resolution order per key:
Resolution order (first-source-wins):
Sources are checked in this order:
kwargs
> service key
> env (AICORE_<NAME>)
> config (AICORE_<NAME> or plain <name>)
> service-like source from JSON in $AICORE_SERVICE_KEY (same structure as a VCAP service object)
falling back to service entry in $VCAP_SERVICES with label 'aicore'
> vcap service key
> default
Important:
- Credentials are extracted from the FIRST source that provides any credential value.
- Values are NOT merged per key across sources. Except resource_group, which is merged.
Warning:
- This function does NOT validate the returned credentials just parsed it from the sources.
- Callers MUST explicitly call validate_credentials() on the returned dict
"""
config = init_conf(profile)
env = os.environ # snapshot for testability
service_like = None
if not config:
# Prefer AICORE_SERVICE_KEY if present; otherwise fall back to the VCAP service.
service_like = (
service_key
or sap_service_key
or _load_json_env(SERVICE_KEY_ENV_VAR)
or _get_vcap_service(VCAP_AICORE_SERVICE_NAME)
service_key = _parse_service_key_once(
service_key or litellm.sap_service_key or os.environ.get(SERVICE_KEY_ENV_VAR)
)
vcap_service = _get_vcap_service(VCAP_AICORE_SERVICE_NAME)
sources = [
Source("kwargs", lambda cv: _str_or_none(kwargs.get(cv.name))),
Source(
"service key",
lambda cv: _resolve_credential_from_service_key(service_key, cv),
),
Source(
"environment variables",
lambda cv: _str_or_none(os.environ.get(f"AICORE_{cv.name.upper()}")),
),
Source(
"config file",
lambda cv: _str_or_none(
config.get(f"AICORE_{cv.name.upper()}")
if config.get(f"AICORE_{cv.name.upper()}") is not None
else config.get(cv.name)
),
),
Source(
"VCAP service",
lambda cv: (
_str_or_none(
_get_nested(
vcap_service,
(("credentials",) + cv.vcap_key) if cv.vcap_key else (cv.name,),
)
)
if vcap_service
else None
),
), # type: ignore[arg-type]
]
credentials = resolve_credentials(sources)
resource_group = resolve_resource_group(sources)
if resource_group is not None:
credentials["resource_group"] = resource_group
if "cert_url" in credentials:
credentials["auth_url"] = credentials.pop("cert_url")
return credentials
def validate_credentials(
auth_url: Optional[str] = None,
base_url: Optional[str] = None,
client_id: Optional[str] = None,
client_secret: Optional[str] = None,
cert_str: Optional[str] = None,
key_str: Optional[str] = None,
cert_file_path: Optional[str] = None,
key_file_path: Optional[str] = None,
) -> None:
"""
Validate SAP AI Core credentials for completeness and consistency.
Args:
auth_url: OAuth2 token endpoint URL (required)
base_url: SAP AI Core API base URL (required)
client_id: OAuth2 client ID (required)
client_secret: OAuth2 client secret (for secret-based auth)
cert_str: PEM-encoded certificate string (for cert-based auth)
key_str: PEM-encoded private key string (for cert-based auth)
cert_file_path: Path to certificate file (for file-based cert auth)
key_file_path: Path to private key file (for file-based cert auth)
Raises:
ValueError: If required fields are missing or authentication mode is ambiguous.
Note:
- This function does NOT validate resource_group (resolved separately).
- Exactly one authentication method must be provided:
* client_secret, OR
* (cert_str AND key_str), OR
* (cert_file_path AND key_file_path)
"""
if not auth_url or not client_id or not base_url:
raise ValueError(
"SAP AI Core credentials not found. "
"Please provide credentials by setting appropriate environment variables "
"(e.g. AICORE_CLIENT_ID, AICORE_CLIENT_SECRET, etc.)"
)
out: Dict[str, str] = {}
for cred in CREDENTIAL_VALUES:
value = _resolve_value(cred, kwargs=kwargs, env=env, config=config, service_like=service_like) # type: ignore
if value is None:
continue
if cred.transform_fn:
value = cred.transform_fn(value)
out[cred.name] = value
if "cert_url" in out.keys():
out["auth_url"] = out.pop("cert_url")
return out
modes = [
bool(client_secret),
bool(cert_str) and bool(key_str),
bool(cert_file_path) and bool(key_file_path),
]
if sum(bool(m) for m in modes) != 1:
raise ValueError(
"SAP AI Core credentials are incomplete. "
"Invalid credentials: provide exactly one of client_secret, "
"(cert_str & key_str), or (cert_file_path & key_file_path)."
)
def _request_token(
client_id: str, auth_url: str, timeout: float, cert_pair=None, client_secret=None
) -> tuple[str, datetime]:
data = {"grant_type": "client_credentials", "client_id": client_id}
if client_secret:
data["client_secret"] = client_secret
resp: Optional[httpx.Response] = None
try:
if cert_pair:
with httpx.Client(cert=cert_pair) as raw_client:
handler = HTTPHandler(client=raw_client)
resp = handler.post(auth_url, data=data, timeout=timeout) # type: ignore[arg-type]
payload = resp.json()
else:
handler = _get_httpx_client()
resp = handler.post(auth_url, data=data, timeout=timeout) # type: ignore[arg-type]
payload = resp.json()
access_token = payload["access_token"]
expires_in = int(payload.get("expires_in", 3600))
expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in)
return f"Bearer {access_token}", expiry_date
except Exception as e:
msg = resp.text if resp is not None else getattr(e, "text", str(e))
raise RuntimeError(f"Token request failed: {msg}") from e
def get_token_creator(
service_key: Optional[str] = None,
service_key: Optional[Union[str, dict]] = None,
profile: Optional[str] = None,
*,
timeout: float = 30.0,
@ -237,7 +423,7 @@ def get_token_creator(
Args:
profile: Optional AICore profile name
timeout: HTTP request timeout in seconds (default 30s)
timeout: Timeout for HTTP requests
expiry_buffer_minutes: Refresh the token this many minutes before expiry
overrides: Any explicit credential overrides (client_id, client_secret, etc.)
@ -251,6 +437,7 @@ def get_token_creator(
)
auth_url = credentials.get("auth_url")
base_url = credentials.get("base_url")
client_id = credentials.get("client_id")
client_secret = credentials.get("client_secret")
cert_str = credentials.get("cert_str")
@ -259,49 +446,30 @@ def get_token_creator(
key_file_path = credentials.get("key_file_path")
# Sanity check
if not auth_url or not client_id:
raise ValueError(
"fetch_credentials did not return valid 'auth_url' or 'client_id'"
)
modes = [
client_secret is not None,
(cert_str is not None and key_str is not None),
(cert_file_path is not None and key_file_path is not None),
]
if sum(bool(m) for m in modes) != 1:
raise ValueError(
"Invalid credentials: provide exactly one of client_secret, "
"(cert_str & key_str), or (cert_file_path & key_file_path)."
)
validate_credentials(
auth_url,
base_url,
client_id,
client_secret,
cert_str,
key_str,
cert_file_path,
key_file_path,
)
lock = Lock()
token: Optional[str] = None
token_expiry: Optional[datetime] = None
def _request_token(cert_pair=None) -> tuple[str, datetime]:
data = {"grant_type": "client_credentials", "client_id": client_id}
if client_secret:
data["client_secret"] = client_secret
client = _get_httpx_client()
# with httpx.Client(cert=cert_pair, timeout=timeout) as client:
resp = client.post(auth_url, data=data)
try:
resp.raise_for_status()
payload = resp.json()
access_token = payload["access_token"]
expires_in = int(payload.get("expires_in", 3600))
expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in)
return f"Bearer {access_token}", expiry_date
except Exception as e:
msg = getattr(resp, "text", str(e))
raise RuntimeError(f"Token request failed: {msg}") from e
def _fetch_token() -> tuple[str, datetime]:
# Case 1: secret-based auth
if client_secret:
return _request_token()
return _request_token(
auth_url=auth_url, # type: ignore[arg-type]
client_id=client_id, # type: ignore[arg-type]
timeout=timeout,
client_secret=client_secret,
)
# Case 2: cert/key strings
if cert_str and key_str:
cert_str_fixed = cert_str.replace("\\n", "\n")
@ -313,9 +481,24 @@ def get_token_creator(
f.write(cert_str_fixed)
with open(key_path, "w") as f:
f.write(key_str_fixed)
return _request_token(cert_pair=(cert_path, key_path))
return _request_token(
auth_url=auth_url, # type: ignore[arg-type]
client_id=client_id, # type: ignore[arg-type]
timeout=timeout,
cert_pair=(cert_path, key_path),
)
# Case 3: file-based cert/key
return _request_token(cert_pair=(cert_file_path, key_file_path))
if cert_file_path is not None and key_file_path is not None:
return _request_token(
auth_url=auth_url, # type: ignore[arg-type]
client_id=client_id, # type: ignore[arg-type]
timeout=timeout,
cert_pair=(cert_file_path, key_file_path),
)
# Defensive guard: should never reach here due to validate_credentials()
raise ValueError(
"Invalid authentication configuration: no valid credentials found. "
)
def get_token() -> str:
nonlocal token, token_expiry

42
litellm/llms/sap/embed/transformation.py
View file

@ -5,6 +5,7 @@ Translates from OpenAI's `/v1/embeddings` to IBM's `/text/embeddings` route.
from typing import Optional, List, Dict, Literal, Union
from pydantic import BaseModel, Field
from functools import cached_property
from litellm.llms.sap.chat.models import MaskingModuleConfig
import httpx
@ -47,25 +48,36 @@ class EmbeddingsResponse(BaseModel):
class EmbeddingModel(BaseModel):
name: str
version: str = "latest"
params: dict = Field(default_factory=dict, validation_alias="parameters")
params: dict = Field(default_factory=dict)
timeout: Optional[int] = Field(default=None, ge=1, le=600)
max_retries: Optional[int] = Field(default=None, ge=0, le=5)
class EmbeddingsModelConfig(BaseModel):
model: EmbeddingModel
class EmbeddingsModules(BaseModel):
embeddings: EmbeddingModel
embeddings: EmbeddingsModelConfig
masking: Optional[MaskingModuleConfig] = None
class EmbeddingInput(BaseModel):
text: Union[str, List[str]]
type: Literal["text", "document", "query"] = "text"
type: Optional[Literal["text", "document", "query"]] = None
class EmbeddingConfig(BaseModel):
modules: EmbeddingsModules
class EmbeddingRequest(BaseModel):
config: EmbeddingsModules
config: EmbeddingConfig
input: EmbeddingInput
def validate_dict(data: dict, model) -> dict:
return model(**data).model_dump()
return model(**data).model_dump(exclude_unset=True, by_alias=True)
class GenAIHubEmbeddingConfig(BaseEmbeddingConfig):
@ -152,15 +164,23 @@ class GenAIHubEmbeddingConfig(BaseEmbeddingConfig):
model_dict["name"] = model
model_dict["version"] = optional_params.get("version", "latest")
model_dict["params"] = optional_params.get("parameters", {})
timeout = optional_params.get("timeout", None)
if timeout is not None:
model_dict["timeout"] = timeout
max_retries = optional_params.get("max_retries", None)
if max_retries is not None:
model_dict["max_retries"] = max_retries
input_dict = {"text": input}
input_type = optional_params.get("type")
if input_type is not None:
input_dict["type"] = input_type
masking = optional_params.get("masking")
masking = {"masking": masking} if masking is not None else {}
body = {
"config": {
"modules": {
"embeddings": {"model": validate_dict(model_dict, EmbeddingModel)}
}
},
"input": validate_dict(input_dict, EmbeddingInput),
"config": {"modules": {"embeddings": {"model": model_dict}, **masking}},
"input": input_dict,
}
body = validate_dict(body, EmbeddingRequest)
return body
def transform_embedding_response(

31
litellm/llms/triton/embedding/transformation.py
View file

@ -8,7 +8,8 @@ from litellm.llms.base_llm.embedding.transformation import (
LiteLLMLoggingObj,
)
from litellm.types.llms.openai import AllEmbeddingInputValues
from litellm.types.utils import EmbeddingResponse
from litellm.types.utils import EmbeddingResponse, Usage
from litellm.utils import token_counter
from ..common_utils import TritonError
@ -103,8 +104,36 @@ class TritonEmbeddingConfig(BaseEmbeddingConfig):
model_response.model = raw_response_json.get("model_name", "None")
model_response.data = _embedding_output
model_response.usage = self._build_embedding_usage(
model=model, request_data=request_data
)
return model_response
def _build_embedding_usage(self, model: str, request_data: dict) -> Usage:
input_data = request_data.get("inputs", [])
input_text_values: List[str] = []
for item in input_data:
if isinstance(item, dict) and item.get("name") == "input_text":
data_values = item.get("data", [])
if isinstance(data_values, list):
input_text_values = [str(value) for value in data_values]
break
prompt_tokens = 0
for text in input_text_values:
if not text:
continue
try:
prompt_tokens += token_counter(model=model, text=text)
except Exception:
prompt_tokens += len(text.split())
return Usage(
prompt_tokens=prompt_tokens,
completion_tokens=0,
total_tokens=prompt_tokens,
)
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
) -> BaseLLMException:

10
litellm/llms/vertex_ai/gemini/transformation.py
View file

@ -763,6 +763,16 @@ def _transform_request_body( # noqa: PLR0915
data["generationConfig"] = generation_config
if cached_content is not None:
data["cachedContent"] = cached_content
if service_tier := optional_params.pop("service_tier", None):
if isinstance(service_tier, str):
if service_tier.lower() == "default":
data["serviceTier"] = "standard"
else:
data["serviceTier"] = service_tier.lower()
else:
data["serviceTier"] = service_tier
# Only add labels for Vertex AI endpoints (not Google GenAI/AI Studio) and only if non-empty
if labels and custom_llm_provider != LlmProviders.GEMINI:
data["labels"] = labels

230
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -318,6 +318,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
"parallel_tool_calls",
"web_search_options",
"include_server_side_tool_invocations",
"service_tier",
]
# Add penalty parameters only for non-preview models
@ -362,6 +363,17 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
"""
return Tools(googleSearch={})
def _map_service_tier_param(self, value: str, optional_params: dict) -> None:
"""
Map OpenAI service_tier (string) to Gemini serviceTier.
'auto' maps to 'priority'.
Other values are passed lowercased.
"""
if value.lower() == "auto":
optional_params["service_tier"] = "priority"
else:
optional_params["service_tier"] = value.lower()
def _transform_computer_use_config(self, computer_use_config: dict) -> dict:
"""
Transform Computer Use configuration to Gemini API format.
@ -1121,6 +1133,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
optional_params = self._add_tools_to_optional_params(
optional_params, [_tools]
)
elif param == "service_tier" and isinstance(value, str):
self._map_service_tier_param(value, optional_params)
elif param == "include_server_side_tool_invocations" and value is True:
optional_params["include_server_side_tool_invocations"] = True
if litellm.vertex_ai_safety_settings is not None:
@ -2415,6 +2429,14 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
"provider_specific_fields", {}
)["traffic_type"] = traffic_type
## ADD SERVICE TIER ##
if getattr(raw_response, "headers", None):
if service_tier := raw_response.headers.get("x-gemini-service-tier"):
if service_tier.lower() == "standard":
setattr(model_response, "service_tier", "default")
else:
setattr(model_response, "service_tier", service_tier.lower())
except Exception as e:
raise VertexAIError(
message="Received={}, Error converting to valid response block={}. File an issue if litellm error - https://github.com/BerriAI/litellm/issues".format(
@ -2513,6 +2535,7 @@ async def make_call(
streaming_response=response.aiter_lines(),
sync_stream=False,
logging_obj=logging_obj,
response_headers=response.headers,
)
# LOGGING
logging_obj.post_call(
@ -2555,6 +2578,7 @@ def make_sync_call(
streaming_response=response.iter_lines(),
sync_stream=True,
logging_obj=logging_obj,
response_headers=response.headers,
)
# LOGGING
@ -3011,7 +3035,11 @@ class VertexLLM(VertexBase):
class ModelResponseIterator:
def __init__(
self, streaming_response, sync_stream: bool, logging_obj: LoggingClass
self,
streaming_response,
sync_stream: bool,
logging_obj: LoggingClass,
response_headers: Optional[Dict[str, str]] = None,
):
from litellm.litellm_core_utils.prompt_templates.common_utils import (
check_is_function_call,
@ -3022,10 +3050,120 @@ class ModelResponseIterator:
self.accumulated_json = ""
self.sent_first_chunk = False
self.logging_obj = logging_obj
self.response_headers = response_headers or {}
self.is_function_call = check_is_function_call(logging_obj)
self.cumulative_tool_call_index: int = 0
self.has_seen_tool_calls: bool = False
def _apply_stream_candidates(
self,
_candidates: List[Candidates],
model_response: Any,
) -> Tuple[List[dict], List[dict], List[dict], List[dict]]:
(
grounding_metadata,
url_context_metadata,
safety_ratings,
citation_metadata,
self.cumulative_tool_call_index,
) = VertexGeminiConfig._process_candidates(
_candidates,
model_response,
self.logging_obj.optional_params,
cumulative_tool_call_index=self.cumulative_tool_call_index,
)
# Track whether tool_calls have been seen across streaming chunks.
# Gemini sends tool_calls and finishReason in separate chunks,
# so we need to remember if earlier chunks contained tool_calls
# to correctly set finish_reason="tool_calls" per the OpenAI spec.
if not self.has_seen_tool_calls:
for choice in model_response.choices:
if (
hasattr(choice, "delta")
and choice.delta
and choice.delta.tool_calls
):
self.has_seen_tool_calls = True
break
# Handle final chunk with finishReason but no content.
# _process_candidates skips candidates without "content",
# so the finish_reason from the final chunk is lost.
if not model_response.choices and _candidates:
from litellm.types.utils import Delta, StreamingChoices
for candidate in _candidates:
finish_reason_str = candidate.get("finishReason")
if finish_reason_str is not None:
if self.has_seen_tool_calls:
mapped_finish_reason = "tool_calls"
else:
mapped_finish_reason = VertexGeminiConfig._check_finish_reason(
None, finish_reason_str
)
choice = StreamingChoices(
finish_reason=mapped_finish_reason,
index=candidate.get("index", 0),
delta=Delta(content=None, role=None),
logprobs=None,
enhancements=None,
)
model_response.choices.append(choice)
# Also handle the case where the final chunk has empty
# content (e.g. text:"") WITH finishReason. In this case
# _process_candidates DOES create a choice, but maps
# finishReason="STOP" to "stop" because the current chunk
# has no tool_calls. Override if we saw tool_calls earlier.
if self.has_seen_tool_calls:
for choice in model_response.choices:
if choice.finish_reason == "stop":
choice.finish_reason = "tool_calls"
setattr(model_response, "vertex_ai_grounding_metadata", grounding_metadata) # type: ignore
setattr(model_response, "vertex_ai_url_context_metadata", url_context_metadata) # type: ignore
setattr(model_response, "vertex_ai_safety_ratings", safety_ratings) # type: ignore
setattr(model_response, "vertex_ai_citation_metadata", citation_metadata) # type: ignore
return grounding_metadata, url_context_metadata, safety_ratings, citation_metadata
def _apply_stream_usage_metadata(
self,
processed_chunk: Any,
model_response: Any,
grounding_metadata: List[dict],
) -> Optional[Usage]:
if "usageMetadata" not in processed_chunk:
return None
usage = VertexGeminiConfig._calculate_usage(
completion_response=processed_chunk,
)
web_search_requests = VertexGeminiConfig._calculate_web_search_requests(
grounding_metadata
)
if web_search_requests is not None:
cast(
PromptTokensDetailsWrapper, usage.prompt_tokens_details
).web_search_requests = web_search_requests
traffic_type = processed_chunk.get("usageMetadata", {}).get("trafficType")
if traffic_type:
model_response._hidden_params.setdefault(
"provider_specific_fields", {}
)["traffic_type"] = traffic_type
service_tier = self.response_headers.get("x-gemini-service-tier")
if service_tier:
if service_tier.lower() == "standard":
setattr(model_response, "service_tier", "default")
else:
setattr(model_response, "service_tier", service_tier.lower())
return usage
def chunk_parser(self, chunk: dict) -> Optional["ModelResponseStream"]:
try:
verbose_logger.debug(f"RAW GEMINI CHUNK: {chunk}")
@ -3043,101 +3181,23 @@ class ModelResponseIterator:
if blocked_response is not None:
model_response = blocked_response
usage: Optional[Usage] = None
_candidates: Optional[List[Candidates]] = processed_chunk.get("candidates")
grounding_metadata: List[dict] = []
url_context_metadata: List[dict] = []
safety_ratings: List[dict] = []
citation_metadata: List[dict] = []
_candidates: Optional[List[Candidates]] = processed_chunk.get("candidates")
if _candidates:
(
grounding_metadata,
url_context_metadata,
safety_ratings,
citation_metadata,
self.cumulative_tool_call_index,
) = VertexGeminiConfig._process_candidates(
_candidates,
model_response,
self.logging_obj.optional_params,
cumulative_tool_call_index=self.cumulative_tool_call_index,
)
) = self._apply_stream_candidates(_candidates, model_response)
# Track whether tool_calls have been seen across streaming chunks.
# Gemini sends tool_calls and finishReason in separate chunks,
# so we need to remember if earlier chunks contained tool_calls
# to correctly set finish_reason="tool_calls" per the OpenAI spec.
if not self.has_seen_tool_calls:
for choice in model_response.choices:
if (
hasattr(choice, "delta")
and choice.delta
and choice.delta.tool_calls
):
self.has_seen_tool_calls = True
break
# Handle final chunk with finishReason but no content.
# _process_candidates skips candidates without "content",
# so the finish_reason from the final chunk is lost.
if not model_response.choices and _candidates:
from litellm.types.utils import Delta, StreamingChoices
for candidate in _candidates:
finish_reason_str = candidate.get("finishReason")
if finish_reason_str is not None:
if self.has_seen_tool_calls:
mapped_finish_reason = "tool_calls"
else:
mapped_finish_reason = (
VertexGeminiConfig._check_finish_reason(
None, finish_reason_str
)
)
choice = StreamingChoices(
finish_reason=mapped_finish_reason,
index=candidate.get("index", 0),
delta=Delta(content=None, role=None),
logprobs=None,
enhancements=None,
)
model_response.choices.append(choice)
# Also handle the case where the final chunk has empty
# content (e.g. text:"") WITH finishReason. In this case
# _process_candidates DOES create a choice, but maps
# finishReason="STOP" to "stop" because the current chunk
# has no tool_calls. Override if we saw tool_calls earlier.
if self.has_seen_tool_calls:
for choice in model_response.choices:
if choice.finish_reason == "stop":
choice.finish_reason = "tool_calls"
setattr(model_response, "vertex_ai_grounding_metadata", grounding_metadata) # type: ignore
setattr(model_response, "vertex_ai_url_context_metadata", url_context_metadata) # type: ignore
setattr(model_response, "vertex_ai_safety_ratings", safety_ratings) # type: ignore
setattr(model_response, "vertex_ai_citation_metadata", citation_metadata) # type: ignore
if "usageMetadata" in processed_chunk:
usage = VertexGeminiConfig._calculate_usage(
completion_response=processed_chunk,
)
web_search_requests = VertexGeminiConfig._calculate_web_search_requests(
grounding_metadata
)
if web_search_requests is not None:
cast(
PromptTokensDetailsWrapper, usage.prompt_tokens_details
).web_search_requests = web_search_requests
traffic_type = processed_chunk.get("usageMetadata", {}).get(
"trafficType"
)
if traffic_type:
model_response._hidden_params.setdefault(
"provider_specific_fields", {}
)["traffic_type"] = traffic_type
usage = self._apply_stream_usage_metadata(
processed_chunk, model_response, grounding_metadata
)
setattr(model_response, "usage", usage) # type: ignore

16
litellm/llms/vertex_ai/vertex_llm_base.py
View file

@ -136,6 +136,11 @@ class VertexBase:
json_obj,
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)
elif isinstance(credential_source, dict) and "executable" in credential_source:
creds = self._credentials_from_pluggable(
json_obj,
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)
else:
creds = self._credentials_from_identity_pool(
json_obj,
@ -190,6 +195,17 @@ class VertexBase:
creds = creds.with_scopes(scopes)
return creds
def _credentials_from_pluggable(self, json_obj, scopes):
try:
from google.auth import pluggable
except ImportError:
raise ImportError(GOOGLE_IMPORT_ERROR_MESSAGE)
creds = pluggable.Credentials.from_info(json_obj)
if scopes and hasattr(creds, "requires_scopes") and creds.requires_scopes:
creds = creds.with_scopes(scopes)
return creds
def _credentials_from_identity_pool_with_aws(self, json_obj, scopes):
try:
from google.auth import aws

189
litellm/model_prices_and_context_window_backup.json
View file

@ -7818,26 +7818,6 @@
"cache_read_input_token_cost": 3e-08,
"cache_creation_input_token_cost": 3.75e-07
},
"bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": {
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 1.8e-05,
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"cache_read_input_token_cost": 3.6e-07,
"cache_creation_input_token_cost": 4.5e-06
},
"bedrock/us-gov-east-1/anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
@ -7856,7 +7836,29 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"supports_native_structured_output": true
},
"bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock",
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_native_structured_output": true
},
"bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": {
"input_cost_per_token": 2.65e-06,
@ -7989,26 +7991,6 @@
"cache_read_input_token_cost": 3e-08,
"cache_creation_input_token_cost": 3.75e-07
},
"bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": {
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 1.8e-05,
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"cache_read_input_token_cost": 3.6e-07,
"cache_creation_input_token_cost": 4.5e-06
},
"bedrock/us-gov-west-1/anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
@ -8027,7 +8009,29 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"supports_native_structured_output": true
},
"bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock",
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_native_structured_output": true
},
"bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": {
"input_cost_per_token": 2.65e-06,
@ -13735,7 +13739,8 @@
"supports_tool_choice": true,
"supports_url_context": true,
"supports_vision": true,
"supports_web_search": true
"supports_web_search": true,
"supports_service_tier": true
},
"gemini-2.5-flash-image": {
"cache_read_input_token_cost": 3e-08,
@ -13784,7 +13789,8 @@
"supports_url_context": true,
"supports_vision": true,
"supports_web_search": false,
"tpm": 8000000
"tpm": 8000000,
"supports_service_tier": true
},
"gemini-3-pro-image-preview": {
"input_cost_per_image": 0.0011,
@ -13818,7 +13824,8 @@
"supports_response_schema": true,
"supports_system_messages": true,
"supports_vision": true,
"supports_web_search": true
"supports_web_search": true,
"supports_service_tier": true
},
"gemini-3.1-flash-image-preview": {
"input_cost_per_image": 0.00056,
@ -13901,7 +13908,8 @@
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true
"supports_native_streaming": true,
"supports_service_tier": true
},
"deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
@ -13980,7 +13988,8 @@
"supports_tool_choice": true,
"supports_url_context": true,
"supports_vision": true,
"supports_web_search": true
"supports_web_search": true,
"supports_service_tier": true
},
"gemini-2.5-flash-lite-preview-09-2025": {
"cache_read_input_token_cost": 1e-08,
@ -14251,7 +14260,8 @@
"supports_tool_choice": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true
"supports_web_search": true,
"supports_service_tier": true
},
"gemini-3-pro-preview": {
"deprecation_date": "2026-03-26",
@ -15033,7 +15043,8 @@
"supports_url_context": true,
"supports_vision": true,
"supports_web_search": true,
"tpm": 8000000
"tpm": 8000000,
"supports_service_tier": true
},
"gemini/gemini-2.5-flash-image": {
"cache_read_input_token_cost": 3e-08,
@ -15083,7 +15094,8 @@
"supports_url_context": true,
"supports_vision": true,
"supports_web_search": true,
"tpm": 8000000
"tpm": 8000000,
"supports_service_tier": true
},
"gemini/gemini-3-pro-image-preview": {
"input_cost_per_image": 0.0011,
@ -15119,7 +15131,8 @@
"supports_response_schema": true,
"supports_system_messages": true,
"supports_vision": true,
"supports_web_search": true
"supports_web_search": true,
"supports_service_tier": true
},
"gemini/gemini-3.1-flash-image-preview": {
"input_cost_per_token": 2.5e-07,
@ -15238,7 +15251,8 @@
"supports_url_context": true,
"supports_vision": true,
"supports_web_search": true,
"tpm": 250000
"tpm": 250000,
"supports_service_tier": true
},
"gemini/gemini-2.5-flash-lite-preview-09-2025": {
"cache_read_input_token_cost": 1e-08,
@ -15678,7 +15692,8 @@
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true,
"tpm": 250000
"tpm": 250000,
"supports_service_tier": true
},
"gemini/gemini-3-flash-preview": {
"cache_read_input_token_cost": 5e-08,
@ -16919,6 +16934,72 @@
"mode": "chat",
"output_cost_per_token": 1.2e-06
},
"baseten/MiniMaxAI/MiniMax-M2.5": {
"input_cost_per_token": 3e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 1.2e-06
},
"baseten/nvidia/Nemotron-120B-A12B": {
"input_cost_per_token": 3e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 7.5e-07
},
"baseten/zai-org/GLM-5": {
"input_cost_per_token": 9.5e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 3.15e-06
},
"baseten/zai-org/GLM-4.7": {
"input_cost_per_token": 6e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 2.2e-06
},
"baseten/zai-org/GLM-4.6": {
"input_cost_per_token": 6e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 2.2e-06
},
"baseten/moonshotai/Kimi-K2.5": {
"input_cost_per_token": 6e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 3e-06
},
"baseten/moonshotai/Kimi-K2-Thinking": {
"input_cost_per_token": 6e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 2.5e-06
},
"baseten/moonshotai/Kimi-K2-Instruct-0905": {
"input_cost_per_token": 6e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 2.5e-06
},
"baseten/openai/gpt-oss-120b": {
"input_cost_per_token": 1e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 5e-07
},
"baseten/deepseek-ai/DeepSeek-V3.1": {
"input_cost_per_token": 5e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 1.5e-06
},
"baseten/deepseek-ai/DeepSeek-V3-0324": {
"input_cost_per_token": 7.7e-07,
"litellm_provider": "baseten",
"mode": "chat",
"output_cost_per_token": 7.7e-07
},
"gmi/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8": {
"input_cost_per_token": 3e-07,
"litellm_provider": "gmi",
@ -38126,4 +38207,4 @@
"supports_native_structured_output": true,
"supports_pdf_input": true
}
}
}

145
litellm/proxy/_experimental/mcp_server/db.py
View file

@ -21,7 +21,9 @@ from litellm.proxy.common_utils.encrypt_decrypt_utils import (
decrypt_value_helper,
encrypt_value_helper,
)
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.proxy.utils import PrismaClient
from litellm.types.llms.custom_http import httpxSpecialProvider
from litellm.types.mcp import MCPCredentials
@ -576,6 +578,7 @@ async def store_user_oauth_credential(
refresh_token: Optional[str] = None,
expires_in: Optional[int] = None,
scopes: Optional[List[str]] = None,
skip_byok_guard: bool = False,
) -> None:
"""Persist an OAuth2 access token for a user+server pair.
@ -604,21 +607,26 @@ async def store_user_oauth_credential(
# Guard against silently overwriting a BYOK credential with an OAuth token.
# BYOK credentials lack a "type" field (or use a non-"oauth2" type).
existing = await prisma_client.db.litellm_mcpusercredentials.find_unique(
where={"user_id_server_id": {"user_id": user_id, "server_id": server_id}}
)
if existing is not None:
_byok_error = ValueError(
f"A non-OAuth2 credential already exists for user {user_id} "
f"and server {server_id}. Refusing to overwrite."
# Skip the guard when the caller knows the row is already an OAuth2 credential
# (e.g. during token refresh), saving an extra DB round-trip.
if not skip_byok_guard:
existing = await prisma_client.db.litellm_mcpusercredentials.find_unique(
where={"user_id_server_id": {"user_id": user_id, "server_id": server_id}}
)
try:
raw = json.loads(base64.urlsafe_b64decode(existing.credential_b64).decode())
except Exception:
# Credential is not base64+JSON — it's a plain-text BYOK key.
raise _byok_error
if raw.get("type") != "oauth2":
raise _byok_error
if existing is not None:
_byok_error = ValueError(
f"A non-OAuth2 credential already exists for user {user_id} "
f"and server {server_id}. Refusing to overwrite."
)
try:
raw = json.loads(
base64.urlsafe_b64decode(existing.credential_b64).decode()
)
except Exception:
# Credential is not base64+JSON — it's a plain-text BYOK key.
raise _byok_error
if raw.get("type") != "oauth2":
raise _byok_error
encoded = base64.urlsafe_b64encode(json.dumps(payload).encode()).decode()
await prisma_client.db.litellm_mcpusercredentials.upsert(
@ -697,6 +705,115 @@ async def list_user_oauth_credentials(
return results
async def refresh_user_oauth_token(
prisma_client: PrismaClient,
user_id: str,
server: Any,
cred: Dict[str, Any],
) -> Optional[Dict[str, Any]]:
"""Attempt to refresh a per-user OAuth2 token using its stored refresh_token.
POSTs to ``server.token_url`` with ``grant_type=refresh_token``.
On success: persists the new credential via ``store_user_oauth_credential``
and returns the updated payload dict.
On failure (network error, invalid_grant, missing refresh_token, …): logs a
warning and returns ``None`` — the caller is responsible for clearing the
stale credential and triggering re-authentication.
"""
refresh_token: Optional[str] = cred.get("refresh_token")
token_url: Optional[str] = getattr(server, "token_url", None)
server_id: str = getattr(server, "server_id", "")
client_id: Optional[str] = getattr(server, "client_id", None)
client_secret: Optional[str] = getattr(server, "client_secret", None)
if not refresh_token:
verbose_proxy_logger.debug(
"refresh_user_oauth_token: no refresh_token stored for user=%s server=%s",
user_id,
server_id,
)
return None
if not token_url:
verbose_proxy_logger.debug(
"refresh_user_oauth_token: server=%s has no token_url configured",
server_id,
)
return None
token_data: Dict[str, str] = {
"grant_type": "refresh_token",
"refresh_token": refresh_token,
}
if client_id:
token_data["client_id"] = client_id
if client_secret:
token_data["client_secret"] = client_secret
try:
async_client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.Oauth2Check
)
response = await async_client.post(
token_url,
headers={"Accept": "application/json"},
data=token_data,
)
response.raise_for_status()
body: Dict[str, Any] = response.json()
except Exception as exc:
verbose_proxy_logger.warning(
"refresh_user_oauth_token: refresh request failed for user=%s server=%s: %s",
user_id,
server_id,
exc,
)
return None
access_token: Optional[str] = body.get("access_token")
if not access_token:
verbose_proxy_logger.warning(
"refresh_user_oauth_token: token response missing access_token for "
"user=%s server=%s",
user_id,
server_id,
)
return None
expires_in: Optional[int] = None
raw_expires = body.get("expires_in")
try:
expires_in = int(raw_expires) if raw_expires is not None else None
except (TypeError, ValueError):
pass
# Rotate refresh token when the provider returns a new one
new_refresh_token: Optional[str] = body.get("refresh_token") or refresh_token
raw_scope = body.get("scope")
scopes: Optional[List[str]] = (
raw_scope.split() if isinstance(raw_scope, str) and raw_scope else None
) or cred.get("scopes")
await store_user_oauth_credential(
prisma_client=prisma_client,
user_id=user_id,
server_id=server_id,
access_token=access_token,
refresh_token=new_refresh_token,
expires_in=expires_in,
scopes=scopes,
skip_byok_guard=True, # Row is already OAuth2; skip the extra find_unique check
)
verbose_proxy_logger.info(
"refresh_user_oauth_token: refreshed token for user=%s server=%s",
user_id,
server_id,
)
return await get_user_oauth_credential(prisma_client, user_id, server_id)
async def approve_mcp_server(
prisma_client: PrismaClient,
server_id: str,

195
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -1,10 +1,11 @@
import json
from typing import Optional
from typing import Any, Dict, Optional
from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
@ -147,6 +148,160 @@ def _resolve_oauth2_server_for_root_endpoints(
return None
def _validate_token_response(
token_response: Dict[str, Any],
validation_rules: Dict[str, Any],
server_id: str,
) -> None:
"""Raise HTTPException 403 if any validation rule doesn't match the token response.
Supports dot-notation for nested fields (e.g. ``"team.enterprise_id"`` checks
``token_response["team"]["enterprise_id"]``). Top-level keys are tried first,
then dot-split traversal. All comparisons are string-coerced so that numeric
values in the response (e.g. ``"org_id": 12345``) match string rules
(``"org_id": "12345"``).
"""
for key, expected in validation_rules.items():
actual: Any = token_response.get(key)
# Try dot-notation traversal when top-level lookup returns None
if actual is None and "." in key:
obj: Any = token_response
for part in key.split("."):
if isinstance(obj, dict):
obj = obj.get(part)
else:
obj = None
break
actual = obj
# Treat absent fields as a distinct failure from a mismatched value
if actual is None:
raise HTTPException(
status_code=403,
detail={
"error": "token_validation_failed",
"server_id": server_id,
"field": key,
"message": (
f"OAuth token rejected: required field '{key}' is absent"
),
},
)
if str(actual) != str(expected):
raise HTTPException(
status_code=403,
detail={
"error": "token_validation_failed",
"server_id": server_id,
"field": key,
"message": (
f"OAuth token rejected: '{key}' = '{actual}', "
f"expected '{expected}'"
),
},
)
async def _extract_user_id_from_request(request: Request) -> Optional[str]:
"""Best-effort extraction of LiteLLM user_id from the request's Authorization header.
Called at the OAuth token endpoint so that per-user tokens can be stored
server-side. Uses a read-only cache lookup to avoid re-running the full
auth pipeline (which has side effects such as rate-limit increments and
spend logging). Returns ``None`` if no cached credential is found.
"""
auth_header = request.headers.get("Authorization") or request.headers.get(
"authorization"
)
if not auth_header:
return None
lower = auth_header.lower()
if not lower.startswith("bearer "):
return None
token = auth_header[7:].strip()
try:
from litellm.proxy._types import hash_token # noqa: PLC0415
from litellm.proxy.proxy_server import user_api_key_cache # noqa: PLC0415
cached = await user_api_key_cache.async_get_cache(hash_token(token))
return getattr(cached, "user_id", None)
except Exception:
return None
async def _store_per_user_token_server_side(
server: MCPServer,
user_id: str,
token_response: Dict[str, Any],
) -> None:
"""Persist the OAuth token server-side and warm the Redis cache.
Called from the token endpoint after a successful code exchange or refresh.
Errors are logged but NOT re-raised — the token is always returned to the
client even when server-side storage fails.
"""
from litellm.proxy._experimental.mcp_server.oauth2_token_cache import ( # noqa: PLC0415
_compute_per_user_token_ttl,
mcp_per_user_token_cache,
)
from litellm.proxy.utils import get_prisma_client_or_throw # noqa: PLC0415
access_token: Optional[str] = token_response.get("access_token")
if not access_token:
return
raw_expires = token_response.get("expires_in")
try:
expires_in: Optional[int] = int(raw_expires) if raw_expires is not None else None
except (TypeError, ValueError):
expires_in = None
refresh_token: Optional[str] = token_response.get("refresh_token") or None
raw_scope = token_response.get("scope")
scopes: Optional[list] = (
raw_scope.split() if isinstance(raw_scope, str) and raw_scope else None
)
try:
prisma_client = get_prisma_client_or_throw(
"Database not connected. Cannot store per-user OAuth token."
)
from litellm.proxy._experimental.mcp_server.db import ( # noqa: PLC0415
store_user_oauth_credential,
)
await store_user_oauth_credential(
prisma_client=prisma_client,
user_id=user_id,
server_id=server.server_id,
access_token=access_token,
refresh_token=refresh_token,
expires_in=expires_in,
scopes=scopes,
)
verbose_logger.info(
"_store_per_user_token_server_side: stored token for user=%s server=%s",
user_id,
server.server_id,
)
except Exception as exc:
verbose_logger.warning(
"_store_per_user_token_server_side: DB storage failed for user=%s server=%s: %s",
user_id,
server.server_id,
exc,
)
return # Don't warm Redis if DB write failed
# Warm the Redis cache so the first subsequent MCP call is a cache hit
ttl = _compute_per_user_token_ttl(server, expires_in)
await mcp_per_user_token_cache.set(
user_id=user_id,
server_id=server.server_id,
access_token=access_token,
ttl=ttl,
)
async def authorize_with_server(
request: Request,
mcp_server: MCPServer,
@ -266,6 +421,44 @@ async def exchange_token_with_server(
token_response = response.json()
access_token = token_response["access_token"]
# Validate token response against server-configured rules before any storage.
# This rejects tokens from wrong Slack workspaces, Atlassian orgs, etc.
if mcp_server.token_validation and isinstance(mcp_server.token_validation, dict):
_validate_token_response(
token_response=token_response,
validation_rules=mcp_server.token_validation,
server_id=mcp_server.server_id,
)
# Store server-side when the server is configured for per-user OAuth and
# the calling client has provided a valid LiteLLM identity.
# Errors are non-fatal: the token is still returned to the client.
if mcp_server.needs_user_oauth_token:
user_id = await _extract_user_id_from_request(request)
if user_id:
try:
await _store_per_user_token_server_side(
server=mcp_server,
user_id=user_id,
token_response=token_response,
)
except Exception as exc:
verbose_logger.warning(
"exchange_token_with_server: server-side storage failed "
"for user=%s server=%s: %s",
user_id,
mcp_server.server_id,
exc,
)
else:
verbose_logger.debug(
"exchange_token_with_server: no LiteLLM user_id found in request; "
"per-user token for server=%s will not be stored server-side. "
"The client should call POST /mcp/server/{id}/oauth-user-credential "
"to store it manually.",
mcp_server.server_id,
)
result = {
"access_token": access_token,
"token_type": token_response.get("token_type", "Bearer"),

31
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -2455,6 +2455,37 @@ class MCPServerManager:
)
tasks.append(during_hook_task)
# For per-user OAuth servers: if the client didn't supply a token in
# oauth2_headers, look up the stored token from Redis / DB. This is the
# call_tool equivalent of _get_user_oauth_extra_headers_from_db used in
# list_tools.
if (
mcp_server.needs_user_oauth_token
and not oauth2_headers
and user_api_key_auth is not None
):
user_id = getattr(user_api_key_auth, "user_id", None)
if user_id:
try:
from litellm.proxy._experimental.mcp_server.server import ( # noqa: PLC0415
_get_user_oauth_extra_headers_from_db,
)
stored_headers = await _get_user_oauth_extra_headers_from_db(
server=mcp_server,
user_api_key_auth=user_api_key_auth,
)
if stored_headers:
oauth2_headers = stored_headers
except Exception as _lookup_exc:
verbose_logger.debug(
"call_tool: per-user token lookup failed for "
"user=%s server=%s: %s",
user_id,
mcp_server.server_id,
_lookup_exc,
)
# For OpenAPI servers, call the tool handler directly instead of via MCP client
if mcp_server.spec_path:
verbose_logger.debug(

108
litellm/proxy/_experimental/mcp_server/oauth2_token_cache.py
View file

@ -17,8 +17,15 @@ from litellm.constants import (
MCP_OAUTH2_TOKEN_CACHE_MAX_SIZE,
MCP_OAUTH2_TOKEN_CACHE_MIN_TTL,
MCP_OAUTH2_TOKEN_EXPIRY_BUFFER_SECONDS,
MCP_PER_USER_TOKEN_DEFAULT_TTL,
MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS,
MCP_PER_USER_TOKEN_REDIS_KEY_PREFIX,
)
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.proxy.common_utils.encrypt_decrypt_utils import (
decrypt_value_helper,
encrypt_value_helper,
)
from litellm.types.llms.custom_http import httpxSpecialProvider
if TYPE_CHECKING:
@ -152,6 +159,107 @@ class MCPOAuth2TokenCache(InMemoryCache):
mcp_oauth2_token_cache = MCPOAuth2TokenCache()
def _compute_per_user_token_ttl(server: "MCPServer", expires_in: Optional[int]) -> int:
"""Compute Redis TTL for a per-user token.
Uses server.token_storage_ttl_seconds when configured; otherwise derives
TTL from expires_in minus the expiry buffer; falls back to the default TTL.
"""
if server.token_storage_ttl_seconds is not None:
return max(server.token_storage_ttl_seconds, 1)
if expires_in is not None:
return max(
expires_in - MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS,
1,
)
return MCP_PER_USER_TOKEN_DEFAULT_TTL
class MCPPerUserTokenCache:
"""Redis-backed cache for per-user OAuth2 access tokens.
Uses LiteLLM's existing ``user_api_key_cache`` (DualCache with optional
Redis backend). Tokens are NaCl-encrypted with ``encrypt_value_helper``
before storage so they are safe at rest in Redis.
Redis key format: ``mcp:per_user_token:{user_id}:{server_id}``
Redis value: ``encrypt_value_helper(access_token)`` — URL-safe base64
"""
def _cache_key(self, user_id: str, server_id: str) -> str:
return f"{MCP_PER_USER_TOKEN_REDIS_KEY_PREFIX}:{user_id}:{server_id}"
async def get(self, user_id: str, server_id: str) -> Optional[str]:
"""Return the plaintext access_token, or None on miss/error."""
try:
from litellm.proxy.proxy_server import user_api_key_cache # noqa: PLC0415
key = self._cache_key(user_id, server_id)
encrypted = await user_api_key_cache.async_get_cache(key)
if encrypted is None:
return None
plaintext = decrypt_value_helper(
encrypted,
key="mcp_per_user_token",
exception_type="debug",
)
return plaintext or None
except Exception as exc:
verbose_logger.debug(
"MCPPerUserTokenCache.get failed for user=%s server=%s: %s",
user_id,
server_id,
exc,
)
return None
async def set(
self,
user_id: str,
server_id: str,
access_token: str,
ttl: int,
) -> None:
"""Store NaCl-encrypted access_token in Redis with the given TTL."""
try:
from litellm.proxy.proxy_server import user_api_key_cache # noqa: PLC0415
key = self._cache_key(user_id, server_id)
encrypted = encrypt_value_helper(access_token)
await user_api_key_cache.async_set_cache(key, encrypted, ttl=ttl)
verbose_logger.debug(
"MCPPerUserTokenCache.set: cached token for user=%s server=%s ttl=%ds",
user_id,
server_id,
ttl,
)
except Exception as exc:
verbose_logger.debug(
"MCPPerUserTokenCache.set failed for user=%s server=%s: %s",
user_id,
server_id,
exc,
)
async def delete(self, user_id: str, server_id: str) -> None:
"""Invalidate the cached token (removes from both in-memory and Redis layers)."""
try:
from litellm.proxy.proxy_server import user_api_key_cache # noqa: PLC0415
key = self._cache_key(user_id, server_id)
await user_api_key_cache.async_delete_cache(key)
except Exception as exc:
verbose_logger.debug(
"MCPPerUserTokenCache.delete failed for user=%s server=%s: %s",
user_id,
server_id,
exc,
)
mcp_per_user_token_cache = MCPPerUserTokenCache()
async def resolve_mcp_auth(
server: "MCPServer",
mcp_auth_header: Optional[Union[str, Dict[str, str]]] = None,

120
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -896,11 +896,17 @@ if MCP_AVAILABLE:
user_api_key_auth: Optional[UserAPIKeyAuth],
prefetched_creds: Optional[Dict[str, Dict[str, Any]]] = None,
) -> Optional[Dict[str, str]]:
"""Look up stored OAuth2 token for (user, server) from DB and return as extra_headers dict.
"""Look up stored OAuth2 token for (user, server) and return as extra_headers dict.
Lookup order:
1. Redis cache (fast path, NaCl-decrypted) — skipped when prefetched_creds supplied
2. prefetched_creds dict (pre-fetched batch DB query) or fresh DB query
3. Auto-refresh when the stored token is expired and a refresh_token exists
Args:
prefetched_creds: Optional dict keyed by server_id with credential payloads.
When provided, avoids a per-server DB round-trip.
When provided, the Redis and individual DB lookups are
skipped in favour of the pre-fetched batch result.
"""
if server.auth_type != MCPAuth.oauth2:
return None
@ -914,8 +920,27 @@ if MCP_AVAILABLE:
from litellm.proxy._experimental.mcp_server.db import ( # noqa: PLC0415
get_user_oauth_credential,
is_oauth_credential_expired,
refresh_user_oauth_token,
)
from litellm.proxy._experimental.mcp_server.oauth2_token_cache import ( # noqa: PLC0415
_compute_per_user_token_ttl,
mcp_per_user_token_cache,
)
# ── Fast path: Redis cache ────────────────────────────────────────
# Only used when prefetched_creds is not supplied (individual lookup).
if prefetched_creds is None:
cached_token = await mcp_per_user_token_cache.get(user_id, server_id)
if cached_token is not None:
verbose_logger.debug(
"_get_user_oauth_extra_headers_from_db: Redis hit for "
"user=%s server=%s",
user_id,
server_id,
)
return {"Authorization": f"Bearer {cached_token}"}
# ── Slow path: DB lookup ──────────────────────────────────────────
if prefetched_creds is not None:
cred = prefetched_creds.get(server_id)
else:
@ -929,18 +954,83 @@ if MCP_AVAILABLE:
cred = await get_user_oauth_credential(
prisma_client, user_id, server_id
)
if cred and cred.get("access_token"):
if is_oauth_credential_expired(cred):
verbose_logger.debug(
f"_get_user_oauth_extra_headers_from_db: token expired for "
f"user={user_id} server={server_id}"
)
if not cred or not cred.get("access_token"):
return None
if is_oauth_credential_expired(cred):
verbose_logger.debug(
"_get_user_oauth_extra_headers_from_db: token expired for "
"user=%s server=%s — attempting refresh",
user_id,
server_id,
)
# Attempt token refresh; requires a DB client (not available from prefetch)
if cred.get("refresh_token"):
try:
from litellm.proxy.utils import ( # noqa: PLC0415
get_prisma_client_or_throw,
)
prisma_client = get_prisma_client_or_throw(
"Database not connected. Cannot refresh OAuth token."
)
cred = await refresh_user_oauth_token(
prisma_client=prisma_client,
user_id=user_id,
server=server,
cred=cred,
)
except Exception as refresh_exc:
verbose_logger.warning(
"_get_user_oauth_extra_headers_from_db: refresh failed "
"for user=%s server=%s: %s",
user_id,
server_id,
refresh_exc,
)
cred = None
if not cred or not cred.get("access_token"):
# Clear stale Redis/cache entry so we don't serve it again.
# Do this for both the individual and prefetch paths so the
# next request doesn't get a stale cache hit.
await mcp_per_user_token_cache.delete(user_id, server_id)
return None
return {"Authorization": f"Bearer {cred['access_token']}"}
access_token: str = cred["access_token"]
# Warm (or re-warm) the Redis cache from the DB result.
# Always write regardless of whether expires_at is present — tokens
# without an expiry are still valid and should be cached using the
# server/default TTL so subsequent requests are fast.
if prefetched_creds is None:
raw_expires = None
expires_at = cred.get("expires_at")
if expires_at:
from datetime import datetime, timezone # noqa: PLC0415
try:
exp_dt = datetime.fromisoformat(expires_at)
if exp_dt.tzinfo is None:
exp_dt = exp_dt.replace(tzinfo=timezone.utc)
remaining = int(
(exp_dt - datetime.now(timezone.utc)).total_seconds()
)
raw_expires = max(remaining, 0) if remaining > 0 else None
except (ValueError, TypeError):
pass
ttl = _compute_per_user_token_ttl(server, raw_expires)
await mcp_per_user_token_cache.set(user_id, server_id, access_token, ttl)
return {"Authorization": f"Bearer {access_token}"}
except Exception as e:
verbose_logger.warning(
f"_get_user_oauth_extra_headers_from_db: failed to retrieve credential for "
f"user={user_id} server={server_id}: {e}"
"_get_user_oauth_extra_headers_from_db: failed to retrieve credential for "
"user=%s server=%s: %s",
user_id,
server_id,
e,
)
return None
@ -2504,6 +2594,14 @@ if MCP_AVAILABLE:
server_name, client_ip=_client_ip
)
if server and server.auth_type == MCPAuth.oauth2 and not oauth2_headers:
# For servers that store per-user tokens server-side, skip the
# pre-emptive 401 — the call_tool / list_tools dispatch will look
# up the stored token from Redis / DB and only fail at the MCP
# protocol level if none is found, giving the client a proper
# tool-execution error rather than an HTTP 401.
if server.needs_user_oauth_token:
continue
request = StarletteRequest(scope)
base_url = get_request_base_url(request)

2
litellm/proxy/_experimental/out/404.html
View file

File diff suppressed because one or more lines are too long

22
litellm/proxy/_experimental/out/__next.__PAGE__.txt
View file

@ -1,28 +1,28 @@
1:"$Sreact.fragment"
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"ClientPageRoot"]
3:I[952683,["/litellm-asset-prefix/_next/static/chunks/9e09de50158b3159.js","/litellm-asset-prefix/_next/static/chunks/7e5fe5584502da06.js","/litellm-asset-prefix/_next/static/chunks/9a17d35f872a6c38.js","/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","/litellm-asset-prefix/_next/static/chunks/99d715502d5069f4.js","/litellm-asset-prefix/_next/static/chunks/7834a5efb7b5f959.js","/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","/litellm-asset-prefix/_next/static/chunks/c847ecdf8c790b0b.js","/litellm-asset-prefix/_next/static/chunks/403c4d96324c23a6.js","/litellm-asset-prefix/_next/static/chunks/a7113797b37526f0.js","/litellm-asset-prefix/_next/static/chunks/ee5f9a39a526e423.js","/litellm-asset-prefix/_next/static/chunks/db0ac43a898048e2.js","/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","/litellm-asset-prefix/_next/static/chunks/c53c9c7afec96700.js","/litellm-asset-prefix/_next/static/chunks/ee9e514b2c2694f7.js","/litellm-asset-prefix/_next/static/chunks/bb71734679762761.js","/litellm-asset-prefix/_next/static/chunks/1fd9dbe73d002173.js","/litellm-asset-prefix/_next/static/chunks/ed901fab61dc16dc.js","/litellm-asset-prefix/_next/static/chunks/ed90bf177ad61e18.js","/litellm-asset-prefix/_next/static/chunks/e1f23fd814ac3500.js","/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","/litellm-asset-prefix/_next/static/chunks/ed4f62880278d987.js","/litellm-asset-prefix/_next/static/chunks/60d899dd52430ef8.js","/litellm-asset-prefix/_next/static/chunks/a5774cdb9f28daa1.js","/litellm-asset-prefix/_next/static/chunks/f04f887c803d9e60.js","/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","/litellm-asset-prefix/_next/static/chunks/a7dc5e0c9d37afe3.js","/litellm-asset-prefix/_next/static/chunks/86819b3a4f820602.js","/litellm-asset-prefix/_next/static/chunks/b6cdb9a433f054f3.js","/litellm-asset-prefix/_next/static/chunks/bf01d87225e5be70.js","/litellm-asset-prefix/_next/static/chunks/b3b05b76472ce110.js","/litellm-asset-prefix/_next/static/chunks/fcdf7322b0aa3e2e.js","/litellm-asset-prefix/_next/static/chunks/47a838c67cdd745e.js","/litellm-asset-prefix/_next/static/chunks/4fc2d71e511309ab.js","/litellm-asset-prefix/_next/static/chunks/0a65da2cd24e2ab6.js","/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","/litellm-asset-prefix/_next/static/chunks/d3ac82723ec9e30d.js","/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","/litellm-asset-prefix/_next/static/chunks/360f35fe2e0a4945.js","/litellm-asset-prefix/_next/static/chunks/0d219667baa010f5.js","/litellm-asset-prefix/_next/static/chunks/7a2dc852f68481ea.js","/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","/litellm-asset-prefix/_next/static/chunks/f9c24d6e7ec43046.js","/litellm-asset-prefix/_next/static/chunks/7c797521435cb59c.js","/litellm-asset-prefix/_next/static/chunks/908828a91f602d8b.js","/litellm-asset-prefix/_next/static/chunks/53ac95bfa383e1b4.js","/litellm-asset-prefix/_next/static/chunks/169b34fe8aeee0c7.js","/litellm-asset-prefix/_next/static/chunks/2e768c2b1dfc8cd5.js"],"default"]
3:I[952683,["/litellm-asset-prefix/_next/static/chunks/9e09de50158b3159.js","/litellm-asset-prefix/_next/static/chunks/7e5fe5584502da06.js","/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","/litellm-asset-prefix/_next/static/chunks/df37a0019220a941.js","/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","/litellm-asset-prefix/_next/static/chunks/99d715502d5069f4.js","/litellm-asset-prefix/_next/static/chunks/7834a5efb7b5f959.js","/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","/litellm-asset-prefix/_next/static/chunks/c847ecdf8c790b0b.js","/litellm-asset-prefix/_next/static/chunks/403c4d96324c23a6.js","/litellm-asset-prefix/_next/static/chunks/a7113797b37526f0.js","/litellm-asset-prefix/_next/static/chunks/ee5f9a39a526e423.js","/litellm-asset-prefix/_next/static/chunks/db0ac43a898048e2.js","/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","/litellm-asset-prefix/_next/static/chunks/c53c9c7afec96700.js","/litellm-asset-prefix/_next/static/chunks/60b0cadba57cd7f7.js","/litellm-asset-prefix/_next/static/chunks/bb71734679762761.js","/litellm-asset-prefix/_next/static/chunks/1fd9dbe73d002173.js","/litellm-asset-prefix/_next/static/chunks/ed901fab61dc16dc.js","/litellm-asset-prefix/_next/static/chunks/ed90bf177ad61e18.js","/litellm-asset-prefix/_next/static/chunks/e1f23fd814ac3500.js","/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","/litellm-asset-prefix/_next/static/chunks/ed4f62880278d987.js","/litellm-asset-prefix/_next/static/chunks/60d899dd52430ef8.js","/litellm-asset-prefix/_next/static/chunks/eaa9f9b9bb3e054b.js","/litellm-asset-prefix/_next/static/chunks/f04f887c803d9e60.js","/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","/litellm-asset-prefix/_next/static/chunks/a02f90f97248b9aa.js","/litellm-asset-prefix/_next/static/chunks/b6cdb9a433f054f3.js","/litellm-asset-prefix/_next/static/chunks/a5774cdb9f28daa1.js","/litellm-asset-prefix/_next/static/chunks/1501e804b4d0f510.js","/litellm-asset-prefix/_next/static/chunks/86819b3a4f820602.js","/litellm-asset-prefix/_next/static/chunks/47a838c67cdd745e.js","/litellm-asset-prefix/_next/static/chunks/be00dd25857a2fb3.js","/litellm-asset-prefix/_next/static/chunks/0a65da2cd24e2ab6.js","/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","/litellm-asset-prefix/_next/static/chunks/d3ac82723ec9e30d.js","/litellm-asset-prefix/_next/static/chunks/7dd16a650b98a4c5.js","/litellm-asset-prefix/_next/static/chunks/bd5cc6a7a48eedc7.js","/litellm-asset-prefix/_next/static/chunks/169b34fe8aeee0c7.js","/litellm-asset-prefix/_next/static/chunks/7a2dc852f68481ea.js","/litellm-asset-prefix/_next/static/chunks/908828a91f602d8b.js","/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","/litellm-asset-prefix/_next/static/chunks/f9c24d6e7ec43046.js","/litellm-asset-prefix/_next/static/chunks/7c797521435cb59c.js","/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","/litellm-asset-prefix/_next/static/chunks/fcdf7322b0aa3e2e.js","/litellm-asset-prefix/_next/static/chunks/53ac95bfa383e1b4.js","/litellm-asset-prefix/_next/static/chunks/0c6c65a34bcde140.js"],"default"]
18:I[897367,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"OutletBoundary"]
19:"$Sreact.suspense"
:HL["/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","style"]
0:{"buildId":"-9iBbUN_ohnDf0d-Ux3Ju","rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/9a17d35f872a6c38.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/99d715502d5069f4.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/7834a5efb7b5f959.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/c847ecdf8c790b0b.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/403c4d96324c23a6.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/a7113797b37526f0.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/ee5f9a39a526e423.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/db0ac43a898048e2.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/c53c9c7afec96700.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/ee9e514b2c2694f7.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/bb71734679762761.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/1fd9dbe73d002173.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/ed901fab61dc16dc.js","async":true}],["$","script","script-18",{"src":"/litellm-asset-prefix/_next/static/chunks/ed90bf177ad61e18.js","async":true}],["$","script","script-19",{"src":"/litellm-asset-prefix/_next/static/chunks/e1f23fd814ac3500.js","async":true}],["$","script","script-20",{"src":"/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","async":true}],["$","script","script-21",{"src":"/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","async":true}],["$","script","script-22",{"src":"/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","async":true}],["$","script","script-23",{"src":"/litellm-asset-prefix/_next/static/chunks/ed4f62880278d987.js","async":true}],["$","script","script-24",{"src":"/litellm-asset-prefix/_next/static/chunks/60d899dd52430ef8.js","async":true}],["$","script","script-25",{"src":"/litellm-asset-prefix/_next/static/chunks/a5774cdb9f28daa1.js","async":true}],["$","script","script-26",{"src":"/litellm-asset-prefix/_next/static/chunks/f04f887c803d9e60.js","async":true}],["$","script","script-27",{"src":"/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","async":true}],["$","script","script-28",{"src":"/litellm-asset-prefix/_next/static/chunks/a7dc5e0c9d37afe3.js","async":true}],["$","script","script-29",{"src":"/litellm-asset-prefix/_next/static/chunks/86819b3a4f820602.js","async":true}],["$","script","script-30",{"src":"/litellm-asset-prefix/_next/static/chunks/b6cdb9a433f054f3.js","async":true}],["$","script","script-31",{"src":"/litellm-asset-prefix/_next/static/chunks/bf01d87225e5be70.js","async":true}],["$","script","script-32",{"src":"/litellm-asset-prefix/_next/static/chunks/b3b05b76472ce110.js","async":true}],["$","script","script-33",{"src":"/litellm-asset-prefix/_next/static/chunks/fcdf7322b0aa3e2e.js","async":true}],"$L6","$L7","$L8","$L9","$La","$Lb","$Lc","$Ld","$Le","$Lf","$L10","$L11","$L12","$L13","$L14","$L15","$L16"],"$L17"]}],"loading":null,"isPartial":false}
0:{"buildId":"lRBQFcrGOsyCYLFEalzGW","rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/df37a0019220a941.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/99d715502d5069f4.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/7834a5efb7b5f959.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/c847ecdf8c790b0b.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/403c4d96324c23a6.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/a7113797b37526f0.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/ee5f9a39a526e423.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/db0ac43a898048e2.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/c53c9c7afec96700.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/60b0cadba57cd7f7.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/bb71734679762761.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/1fd9dbe73d002173.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/ed901fab61dc16dc.js","async":true}],["$","script","script-18",{"src":"/litellm-asset-prefix/_next/static/chunks/ed90bf177ad61e18.js","async":true}],["$","script","script-19",{"src":"/litellm-asset-prefix/_next/static/chunks/e1f23fd814ac3500.js","async":true}],["$","script","script-20",{"src":"/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","async":true}],["$","script","script-21",{"src":"/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","async":true}],["$","script","script-22",{"src":"/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","async":true}],["$","script","script-23",{"src":"/litellm-asset-prefix/_next/static/chunks/ed4f62880278d987.js","async":true}],["$","script","script-24",{"src":"/litellm-asset-prefix/_next/static/chunks/60d899dd52430ef8.js","async":true}],["$","script","script-25",{"src":"/litellm-asset-prefix/_next/static/chunks/eaa9f9b9bb3e054b.js","async":true}],["$","script","script-26",{"src":"/litellm-asset-prefix/_next/static/chunks/f04f887c803d9e60.js","async":true}],["$","script","script-27",{"src":"/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","async":true}],["$","script","script-28",{"src":"/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","async":true}],["$","script","script-29",{"src":"/litellm-asset-prefix/_next/static/chunks/a02f90f97248b9aa.js","async":true}],["$","script","script-30",{"src":"/litellm-asset-prefix/_next/static/chunks/b6cdb9a433f054f3.js","async":true}],["$","script","script-31",{"src":"/litellm-asset-prefix/_next/static/chunks/a5774cdb9f28daa1.js","async":true}],["$","script","script-32",{"src":"/litellm-asset-prefix/_next/static/chunks/1501e804b4d0f510.js","async":true}],["$","script","script-33",{"src":"/litellm-asset-prefix/_next/static/chunks/86819b3a4f820602.js","async":true}],"$L6","$L7","$L8","$L9","$La","$Lb","$Lc","$Ld","$Le","$Lf","$L10","$L11","$L12","$L13","$L14","$L15","$L16"],"$L17"]}],"loading":null,"isPartial":false}
4:{}
5:"$0:rsc:props:children:0:props:serverProvidedParams:params"
6:["$","script","script-34",{"src":"/litellm-asset-prefix/_next/static/chunks/47a838c67cdd745e.js","async":true}]
7:["$","script","script-35",{"src":"/litellm-asset-prefix/_next/static/chunks/4fc2d71e511309ab.js","async":true}]
7:["$","script","script-35",{"src":"/litellm-asset-prefix/_next/static/chunks/be00dd25857a2fb3.js","async":true}]
8:["$","script","script-36",{"src":"/litellm-asset-prefix/_next/static/chunks/0a65da2cd24e2ab6.js","async":true}]
9:["$","script","script-37",{"src":"/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","async":true}]
a:["$","script","script-38",{"src":"/litellm-asset-prefix/_next/static/chunks/d3ac82723ec9e30d.js","async":true}]
b:["$","script","script-39",{"src":"/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","async":true}]
c:["$","script","script-40",{"src":"/litellm-asset-prefix/_next/static/chunks/360f35fe2e0a4945.js","async":true}]
d:["$","script","script-41",{"src":"/litellm-asset-prefix/_next/static/chunks/0d219667baa010f5.js","async":true}]
b:["$","script","script-39",{"src":"/litellm-asset-prefix/_next/static/chunks/7dd16a650b98a4c5.js","async":true}]
c:["$","script","script-40",{"src":"/litellm-asset-prefix/_next/static/chunks/bd5cc6a7a48eedc7.js","async":true}]
d:["$","script","script-41",{"src":"/litellm-asset-prefix/_next/static/chunks/169b34fe8aeee0c7.js","async":true}]
e:["$","script","script-42",{"src":"/litellm-asset-prefix/_next/static/chunks/7a2dc852f68481ea.js","async":true}]
f:["$","script","script-43",{"src":"/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","async":true}]
f:["$","script","script-43",{"src":"/litellm-asset-prefix/_next/static/chunks/908828a91f602d8b.js","async":true}]
10:["$","script","script-44",{"src":"/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","async":true}]
11:["$","script","script-45",{"src":"/litellm-asset-prefix/_next/static/chunks/f9c24d6e7ec43046.js","async":true}]
12:["$","script","script-46",{"src":"/litellm-asset-prefix/_next/static/chunks/7c797521435cb59c.js","async":true}]
13:["$","script","script-47",{"src":"/litellm-asset-prefix/_next/static/chunks/908828a91f602d8b.js","async":true}]
14:["$","script","script-48",{"src":"/litellm-asset-prefix/_next/static/chunks/53ac95bfa383e1b4.js","async":true}]
15:["$","script","script-49",{"src":"/litellm-asset-prefix/_next/static/chunks/169b34fe8aeee0c7.js","async":true}]
16:["$","script","script-50",{"src":"/litellm-asset-prefix/_next/static/chunks/2e768c2b1dfc8cd5.js","async":true}]
13:["$","script","script-47",{"src":"/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","async":true}]
14:["$","script","script-48",{"src":"/litellm-asset-prefix/_next/static/chunks/fcdf7322b0aa3e2e.js","async":true}]
15:["$","script","script-49",{"src":"/litellm-asset-prefix/_next/static/chunks/53ac95bfa383e1b4.js","async":true}]
16:["$","script","script-50",{"src":"/litellm-asset-prefix/_next/static/chunks/0c6c65a34bcde140.js","async":true}]
17:["$","$L18",null,{"children":["$","$19",null,{"name":"Next.MetadataOutlet","children":"$@1a"}]}]
1a:null

34
litellm/proxy/_experimental/out/__next._full.txt
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/__next._head.txt
View file

@ -3,4 +3,4 @@
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"MetadataBoundary"]
4:"$Sreact.suspense"
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"IconMark"]
0:{"buildId":"-9iBbUN_ohnDf0d-Ux3Ju","rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.1d32c690.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"./favicon.ico"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"loading":null,"isPartial":false}
0:{"buildId":"lRBQFcrGOsyCYLFEalzGW","rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.1d32c690.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"./favicon.ico"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"loading":null,"isPartial":false}

2
litellm/proxy/_experimental/out/__next._index.txt
View file

@ -5,4 +5,4 @@
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"default"]
:HL["/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/c3f387b3358b56db.css","style"]
0:{"buildId":"-9iBbUN_ohnDf0d-Ux3Ju","rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/c3f387b3358b56db.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/9e09de50158b3159.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/7e5fe5584502da06.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]]}],"loading":null,"isPartial":false}
0:{"buildId":"lRBQFcrGOsyCYLFEalzGW","rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/c3f387b3358b56db.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/9e09de50158b3159.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/7e5fe5584502da06.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]]}],"loading":null,"isPartial":false}

2
litellm/proxy/_experimental/out/__next._tree.txt
View file

@ -2,4 +2,4 @@
:HL["/litellm-asset-prefix/_next/static/chunks/c3f387b3358b56db.css","style"]
:HL["/litellm-asset-prefix/_next/static/media/83afe278b6a6bb3c-s.p.3a6ba036.woff2","font",{"crossOrigin":"","type":"font/woff2"}]
:HL["/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","style"]
0:{"buildId":"-9iBbUN_ohnDf0d-Ux3Ju","tree":{"name":"","paramType":null,"paramKey":"","hasRuntimePrefetch":false,"slots":{"children":{"name":"__PAGE__","paramType":null,"paramKey":"__PAGE__","hasRuntimePrefetch":false,"slots":null,"isRootLayout":false}},"isRootLayout":true},"staleTime":300}
0:{"buildId":"lRBQFcrGOsyCYLFEalzGW","tree":{"name":"","paramType":null,"paramKey":"","hasRuntimePrefetch":false,"slots":{"children":{"name":"__PAGE__","paramType":null,"paramKey":"__PAGE__","hasRuntimePrefetch":false,"slots":null,"isRootLayout":false}},"isRootLayout":true},"staleTime":300}

1
litellm/proxy/_experimental/out/_next/static/chunks/086f1dd580fe748e.js Normal file
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/2e768c2b1dfc8cd5.js → litellm/proxy/_experimental/out/_next/static/chunks/0c6c65a34bcde140.js
View file

File diff suppressed because one or more lines are too long

91
litellm/proxy/_experimental/out/_next/static/chunks/0d219667baa010f5.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/11c5483d145114d0.js Normal file
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/161a2ab7f4e973ca.js → litellm/proxy/_experimental/out/_next/static/chunks/130cfc006c4f7d77.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/13670846207c3e16.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1379bf26a33536ad.js Normal file
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more