mirror of https://github.com/BerriAI/litellm.git synced 2026-09-05 08:07:05 +00:00

feat(terraform): vendor terraform-provider-litellm as source of truth with endpoint drift CI (#32241 )

* feat(terraform): vendor terraform-provider-litellm as source of truth with endpoint drift CI

* fix(terraform): address review feedback on vendored provider

Replace deprecated io/ioutil with io. Remove the unused org/team CRUD
client methods so the endpoint audit only tracks live call sites
(54 -> 46). Redact request/response logs by parsing the JSON and
recursively masking sensitive fields, which fixes the nested-object
leak in the old credential_values regex, with a regex fallback for
non-JSON payloads; covered by new unit tests. Docs: stop showing
api_key inside vector store litellm_params and document that Sensitive
attributes still persist in plaintext state, recommending
litellm_credential_name and an encrypted state backend.

* fix(terraform): stop persisting server-returned litellm_params into vector store state

The vector store Read wrote litellm_params straight back from the API
response into state. The proxy redacts secrets in those responses, so
the readback overwrote user config with redaction sentinels and caused
perpetual diffs, and against a server that returns raw values it would
persist secrets into a non-Sensitive attribute. Read now preserves the
config value like the credential and model resources do, litellm_params
is marked Sensitive, and a regression test pins that a server-returned
api_key never lands in state

* fix(terraform): send role on team member update and stop persisting server env into MCP state

The team member update payload omitted role, and the proxy leaves role
unchanged when the field is absent, so a role downgrade reported as
applied by Terraform never took effect on the proxy. The update now
always sends the configured role (the attribute is Required).

The MCP server resource wrote env straight back from API responses
into a non-Sensitive attribute, pulling admin-visible secrets into
state and, for sanitized responses, blanking user config. Read now
preserves the config value, env is marked Sensitive, and the docs warn
against passing secrets via args. Regression tests cover both fixes
and fail against the previous behavior.

2026-07-07 09:16:59 -07:00

9.6 KiB

Raw Blame History

litellm_model Resource

Manages a LiteLLM model configuration. This resource allows you to create, update, and delete model configurations in your LiteLLM instance.

Example Usage

Basic OpenAI Model

resource "litellm_model" "gpt4" {
  model_name          = "gpt-4-proxy"
  custom_llm_provider = "openai"
  model_api_key       = var.openai_api_key
  base_model          = "gpt-4"
  tier                = "paid"
  mode                = "chat"
  
  input_cost_per_million_tokens  = 30.0
  output_cost_per_million_tokens = 60.0
}

Advanced Model with All Features

resource "litellm_model" "advanced_gpt4" {
  model_name          = "gpt-4-advanced"
  custom_llm_provider = "openai"
  model_api_key       = var.openai_api_key
  model_api_base      = "https://api.openai.com/v1"
  api_version         = "2023-05-15"
  base_model          = "gpt-4"
  tier                = "paid"
  team_id             = "team-123"
  mode                = "chat"
  reasoning_effort    = "medium"
  thinking_enabled    = true
  thinking_budget_tokens = 1024
  merge_reasoning_content_in_choices = true
  tpm                 = 100000
  rpm                 = 1000
  
  # Cost configuration (per million tokens)
  input_cost_per_million_tokens  = 30.0    # $0.03 per 1k tokens = $30 per million
  output_cost_per_million_tokens = 60.0    # $0.06 per 1k tokens = $60 per million
}

AWS Bedrock Model with Cross-Account Access

resource "litellm_model" "bedrock_claude" {
  model_name          = "bedrock-claude-proxy"
  custom_llm_provider = "bedrock"
  base_model          = "anthropic.claude-3-sonnet-20240229-v1:0"
  tier                = "paid"
  mode                = "chat"
  
  # AWS configuration with cross-account access
  aws_access_key_id     = var.aws_access_key_id
  aws_secret_access_key = var.aws_secret_access_key
  aws_region_name       = "us-east-1"
  aws_session_name      = "litellm-cross-account-session"
  aws_role_name         = "arn:aws:iam::123456789012:role/LiteLLMCrossAccountRole"
  
  input_cost_per_million_tokens  = 3.0
  output_cost_per_million_tokens = 15.0
}

Anthropic Model

resource "litellm_model" "claude" {
  model_name          = "claude-proxy"
  custom_llm_provider = "anthropic"
  model_api_key       = var.anthropic_api_key
  base_model          = "claude-3-sonnet-20240229"
  tier                = "paid"
  mode                = "chat"
  
  input_cost_per_million_tokens  = 3.0
  output_cost_per_million_tokens = 15.0
}

Azure OpenAI Model

resource "litellm_model" "azure_gpt4" {
  model_name          = "azure-gpt4-proxy"
  custom_llm_provider = "azure"
  model_api_key       = var.azure_openai_key
  model_api_base      = var.azure_openai_endpoint
  api_version         = "2023-12-01-preview"
  base_model          = "gpt-4"
  tier                = "paid"
  mode                = "chat"
  
  input_cost_per_million_tokens  = 30.0
  output_cost_per_million_tokens = 60.0
}

Argument Reference

The following arguments are supported:

model_name - (Required) string. The name of the model configuration used to identify the model in API calls.
custom_llm_provider - (Required) string. The LLM provider for this model (e.g., "openai", "anthropic", "azure", "bedrock").
model_api_key - (Optional) string (Sensitive). The API key for the underlying model provider. Sensitive attributes are hidden from Terraform output but still stored in plaintext in the state file; prefer storing provider secrets in a litellm_credential and referencing it via litellm_credential_name, and secure your state backend.
model_api_base - (Optional) string. The base URL for the model provider's API.
api_version - (Optional) string. The API version to use for the model provider.
base_model - (Required) string. The actual model identifier from the provider (e.g., "gpt-4", "claude-2").
litellm_credential_name - (Optional) string. Name of a LiteLLM credential to use for this model.
tier - (Optional) string. The usage tier for this model. Valid values are "free" or "paid". Default: "free".
team_id - (Optional) string. Associate the model with a specific team.
mode - (Optional) string. The intended use of the model. Valid values are:
- completion
- embedding
- image_generation
- chat
- moderation
- audio_transcription
- audio_speech
- rerank
tpm - (Optional) integer. Tokens per minute limit for this model.
rpm - (Optional) integer. Requests per minute limit for this model.
reasoning_effort - (Optional) string. Configures the model's reasoning effort level. Valid values are:
- low
- medium
- high
thinking_enabled - (Optional) boolean. Enables the model's thinking capability. Default: false.
thinking_budget_tokens - (Optional) integer. Sets the token budget for the model's thinking capability. Default: 1024. Note: this field is only relevant when thinking_enabled = true.
merge_reasoning_content_in_choices - (Optional) boolean. When set to true, merges reasoning content into the model's choices.
input_cost_per_million_tokens - (Optional) float. Cost per million input tokens. The provider converts this to a per-token cost sent to the API.
output_cost_per_million_tokens - (Optional) float. Cost per million output tokens. The provider converts this to a per-token cost sent to the API.
input_cost_per_pixel - (Optional) float. Cost applied per input pixel for models that charge by image size.
output_cost_per_pixel - (Optional) float. Cost applied per output pixel for image-generation models.
input_cost_per_second - (Optional) float. Cost applied per input second for audio/transcription models.
output_cost_per_second - (Optional) float. Cost applied per output second for audio/transcription models.
vertex_project - (Optional) string. Vertex AI project id (for custom_llm_provider = "vertex").
vertex_location - (Optional) string. Vertex AI location (e.g., us-central1).
vertex_credentials - (Optional) string. Vertex credentials (JSON string or path depending on your setup).
additional_litellm_params - (Optional) map(string). A map of arbitrary additional parameters that will be merged into the litellm_params object sent to the LiteLLM API. This is intended for provider-specific or experimental options not exposed as dedicated arguments.

Conversion and behavior rules (how the provider handles values):
- When values in the map are strings the provider will attempt to coerce them:
  - "true" / "false" (strings) -> boolean true / false
  - Numeric strings are parsed first as integers; if integer parsing fails, parsed as floats (e.g., "16384" -> 16384, "0.75" -> 0.75)
  - JSON strings (starting with [ or {) are parsed as JSON objects/arrays
  - Non-convertible strings remain strings
- Non-string map values (if supplied) are passed through unchanged.
- The provider merges these keys into the litellm_params payload sent to the API.
- Note: the remote API may not echo back all custom parameters; this provider preserves additional_litellm_params in state when present in configuration.
Special parameter: additional_drop_params
- When additional_drop_params is provided as a JSON array string, it specifies parameters to remove from the final litellm_params before sending to the API
- This allows you to override or remove built-in parameters if needed
- The additional_drop_params key itself is not included in the final parameters
Example showing booleans, integers, floats, strings, and parameter dropping:
```
resource "litellm_model" "with_additional" {
  model_name          = "custom-model"
  custom_llm_provider = "openai"
  model_api_key       = var.openai_api_key
  base_model          = "gpt-4"
  mode                = "chat"

  additional_litellm_params = {
    "use_fine_tune"          = "true"                    # becomes boolean true
    "max_context"            = "16384"                   # becomes integer 16384
    "scale"                  = "0.75"                    # becomes float 0.75
    "note"                   = "for testing"             # stays string
    "complex_config"         = "{\"nested\": {\"value\": 42}}"  # parsed as JSON object
    "additional_drop_params" = "[\"reasoningEffort\"]"   # removes reasoningEffort parameter
  }
}
```

AWS-specific Configuration

aws_access_key_id - (Optional) string (Sensitive). AWS access key ID for AWS-based models.
aws_secret_access_key - (Optional) string (Sensitive). AWS secret access key for AWS-based models. As with model_api_key, the value is stored in plaintext in the state file; prefer a litellm_credential referenced via litellm_credential_name and secure your state backend.
aws_region_name - (Optional) string. AWS region name for AWS-based models.
aws_session_name - (Optional) string (Sensitive). AWS session name for cross-account access scenarios.
aws_role_name - (Optional) string (Sensitive). AWS IAM role name for cross-account access scenarios.

Attribute Reference

In addition to the arguments above, the following attributes are exported:

id - The ID of the model configuration.

Import

Model configurations can be imported using the model ID:

terraform import litellm_model.gpt4 <model-id>

Note: The model ID is generated when the model is created and is different from the model_name.

Security Note

When using this resource, ensure that sensitive information such as API keys and AWS credentials are stored securely. It's recommended to use environment variables or a secure secret management solution rather than hardcoding these values in your Terraform configuration files.

9.6 KiB Raw Blame History