litellm/AGENTS.md

INSTRUCTIONS FOR LITELLM

This document provides comprehensive instructions for AI agents working in the LiteLLM repository.

OVERVIEW

LiteLLM is a unified interface for 100+ LLMs that:

• Translates inputs to provider-specific completion, embedding, and image generation endpoints
• Provides consistent OpenAI-format output across all providers
• Includes retry/fallback logic across multiple deployments (Router)
• Offers a proxy server (LLM Gateway) with budgets, rate limits, and authentication
• Supports advanced features like function calling, streaming, caching, and observability

REPOSITORY STRUCTURE

Core Components

• litellm/ - Main library code
  • llms/ - Provider-specific implementations (OpenAI, Anthropic, Azure, etc.)
  • proxy/ - Proxy server implementation (LLM Gateway)
  • router_utils/ - Load balancing and fallback logic
  • types/ - Type definitions and schemas
  • integrations/ - Third-party integrations (observability, caching, etc.)

Key Directories

• tests/ - Comprehensive test suites
• docs/my-website/ - Documentation website
• ui/litellm-dashboard/ - Admin dashboard UI
• enterprise/ - Enterprise-specific features

DEVELOPMENT GUIDELINES

MAKING CODE CHANGES

1. Provider Implementations: When adding/modifying LLM providers:

   • Follow existing patterns in litellm/llms/{provider}/
   • Implement proper transformation classes that inherit from BaseConfig
   • Support both sync and async operations
   • Handle streaming responses appropriately
   • Include proper error handling with provider-specific exceptions

2. Type Safety:

   • Use proper type hints throughout
   • Update type definitions in litellm/types/
   • Ensure compatibility with both Pydantic v1 and v2

3. Testing:

   • Add tests in appropriate tests/ subdirectories
   • Include both unit tests and integration tests
   • Test provider-specific functionality thoroughly
   • Consider adding load tests for performance-critical changes

MAKING CODE CHANGES FOR THE UI (IGNORE FOR BACKEND)

1. Tremor is DEPRECATED, do not use Tremor components in new features/changes

   • The only exception is the Tremor Table component and its required Tremor Table sub components.

2. Use Common Components as much as possible:

   • These are usually defined in the common_components directory
   • Use these components as much as possible and avoid building new components unless needed

3. Testing:

   • The codebase uses Vitest and React Testing Library
   • Query Priority Order: Use query methods in this order: getByRole, getByLabelText, getByPlaceholderText, getByText, getByTestId
   • Always use screen instead of destructuring from render() (e.g., use screen.getByText() not getByText)
   • Wrap user interactions in act(): Always wrap fireEvent calls with act() to ensure React state updates are properly handled
   • Use query methods for absence checks: Use queryBy* methods (not getBy*) when expecting an element to NOT be present
   • Test names must start with "should": All test names should follow the pattern it("should ...")
   • Mock external dependencies: Check setupTests.ts for global mocks and mock child components/networking calls as needed
   • Structure tests properly:
     • First test should verify the component renders successfully
     • Subsequent tests should focus on functionality and user interactions
     • Use waitFor for async operations that aren't already awaited
   • Avoid using querySelector: Prefer React Testing Library queries over direct DOM manipulation

IMPORTANT PATTERNS

1. Function/Tool Calling:

   • LiteLLM standardizes tool calling across providers
   • OpenAI format is the standard, with transformations for other providers
   • See litellm/llms/anthropic/chat/transformation.py for complex tool handling

2. Streaming:

   • All providers should support streaming where possible
   • Use consistent chunk formatting across providers
   • Handle both sync and async streaming

3. Error Handling:

   • Use provider-specific exception classes
   • Maintain consistent error formats across providers
   • Include proper retry logic and fallback mechanisms

4. Configuration:

   • Support both environment variables and programmatic configuration
   • Use BaseConfig classes for provider configurations
   • Allow dynamic parameter passing

PROXY SERVER (LLM GATEWAY)

The proxy server is a critical component that provides:

• Authentication and authorization
• Rate limiting and budget management
• Load balancing across multiple models/deployments
• Observability and logging
• Admin dashboard UI
• Enterprise features

Key files:

• litellm/proxy/proxy_server.py - Main server implementation
• litellm/proxy/auth/ - Authentication logic
• litellm/proxy/management_endpoints/ - Admin API endpoints

MCP (MODEL CONTEXT PROTOCOL) SUPPORT

LiteLLM supports MCP for agent workflows:

• MCP server integration for tool calling
• Transformation between OpenAI and MCP tool formats
• Support for external MCP servers (Zapier, Jira, Linear, etc.)
• See litellm/experimental_mcp_client/ and litellm/proxy/_experimental/mcp_server/

RUNNING SCRIPTS

Use poetry run python script.py to run Python scripts in the project environment (for non-test files).

GITHUB TEMPLATES

When opening issues or pull requests, follow these templates:

Bug Reports (.github/ISSUE_TEMPLATE/bug_report.yml)

• Describe what happened vs. expected behavior
• Include relevant log output
• Specify LiteLLM version
• Indicate if you're part of an ML Ops team (helps with prioritization)

Feature Requests (.github/ISSUE_TEMPLATE/feature_request.yml)

• Clearly describe the feature
• Explain motivation and use case with concrete examples

Pull Requests (.github/pull_request_template.md)

• Add at least 1 test in tests/litellm/
• Ensure make test-unit passes

TESTING CONSIDERATIONS

1. Provider Tests: Test against real provider APIs when possible
2. Proxy Tests: Include authentication, rate limiting, and routing tests
3. Performance Tests: Load testing for high-throughput scenarios
4. Integration Tests: End-to-end workflows including tool calling

DOCUMENTATION

• Keep documentation in sync with code changes
• Update provider documentation when adding new providers
• Include code examples for new features
• Update changelog and release notes

SECURITY CONSIDERATIONS

• Handle API keys securely
• Validate all inputs, especially for proxy endpoints
• Consider rate limiting and abuse prevention
• Follow security best practices for authentication

ENTERPRISE FEATURES

• Some features are enterprise-only
• Check enterprise/ directory for enterprise-specific code
• Maintain compatibility between open-source and enterprise versions

COMMON PITFALLS TO AVOID

1. Breaking Changes: LiteLLM has many users - avoid breaking existing APIs
2. Provider Specifics: Each provider has unique quirks - handle them properly
3. Rate Limits: Respect provider rate limits in tests
4. Memory Usage: Be mindful of memory usage in streaming scenarios
5. Dependencies: Keep dependencies minimal and well-justified

HELPFUL RESOURCES

• Main documentation: https://docs.litellm.ai/
• Provider-specific docs in docs/my-website/docs/providers/
• Admin UI for testing proxy features

WHEN IN DOUBT

• Follow existing patterns in the codebase
• Check similar provider implementations
• Ensure comprehensive test coverage
• Update documentation appropriately
• Consider backward compatibility impact

Cursor Cloud specific instructions

Services overview

Service	How to run	Notes
LiteLLM Proxy (backend)	DATABASE_URL="$DATABASE_URL" LITELLM_MASTER_KEY="$LITELLM_MASTER_KEY" STORE_MODEL_IN_DB="True" poetry run litellm --port 4000	Requires PostgreSQL running and Prisma generated. See .env.example for default credential values.
Dashboard UI (frontend)	cd ui/litellm-dashboard && npm run dev	Next.js dev server on port 3000; auto-proxies API calls to http://localhost:4000 in dev mode
PostgreSQL	sudo pg_ctlcluster 16 main start	DB name: litellm. Credentials in .env.example.

Setup prerequisites (already done by update script)

• poetry install --with dev,proxy-dev --extras proxy — installs all Python deps
• npm install in ui/litellm-dashboard/ — installs UI deps

Non-obvious caveats

• Prisma must be generated before the proxy starts. Run poetry run prisma generate --schema=./litellm/proxy/schema.prisma and DATABASE_URL=<your_url> poetry run prisma db push --schema=./litellm/proxy/schema.prisma --accept-data-loss once after DB setup.
• UI login: username admin, password is the LITELLM_MASTER_KEY env var value.
• The Next.js dev server at port 3000 uses networking.tsx which defaults proxyBaseUrl to http://localhost:4000 when NODE_ENV=development. Do not access the proxy's built-in UI at port 4000 for dev work; use port 3000.
• UI tests use Vitest: cd ui/litellm-dashboard && npm test. Python tests: poetry run pytest tests/test_litellm/ -x -v.
• Ruff/Black/MyPy lint commands are in the Makefile — see make lint, make format.
• poetry must be on PATH; the update script installs it to ~/.local/bin.
• Poetry 2.x is used (lock file is Poetry 2.3.2 format).