This module creates a Confluent Cloud Kafka cluster on GCP and stores API Keys and Secrets in GCP Secret Manager.

The module is split into two submodules for better separation of concerns:

tf-module-gcp-confluent/
├── modules/
│   ├── network/           # Environment, private networking, bastion, Schema Registry
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── providers.tf
│   └── cluster/           # Kafka cluster, API keys, secrets, cluster link
│       ├── main.tf
│       ├── variables.tf
│       ├── outputs.tf
│       └── providers.tf
├── main.tf                # Root module that composes both submodules
├── vars.tf                # Passthrough variables
├── outputs.tf             # Passthrough outputs
└── providers.tf           # Provider configuration

Manages environment-level and networking resources:

• Confluent environment (conditional creation)
• Private Service Connect network and access
• Private Link Attachment with GCP compute/DNS resources
• Bastion host for SSH tunneling
• Schema Registry service account and API key

Manages cluster-level resources:

• Kafka cluster
• Kafka service account and API key
• GCP Secret Manager secrets
• Cluster link and mirror topics

The root module composes both submodules while maintaining backward compatibility with existing configurations.

This module uses Confluent provider v2.x which has breaking changes:

• Schema Registry: Now auto-provisioned with environments. The schema_registry_region and package variables have been removed.
• Cluster Type: The project_env variable has been removed. Use cluster_type instead.
• Terraform Version: Requires Terraform >= 1.5.7

1. Update your Terraform version to >= 1.5.7
2. Remove schema_registry_region and package from your module calls
3. Replace project_env with cluster_type:
   • project_env = "staging" → cluster_type = "basic"
   • project_env = "prod" → cluster_type = "standard"
4. Run terraform state rm for the Schema Registry cluster resource before applying (it's now a data source)

GCP Secret Manager secrets include the cluster type in their names to avoid collisions when multiple clusters exist in the same project. Secret names follow this pattern:

kafka_cluster_api_key_<cluster_type>
kafka_cluster_api_secret_<cluster_type>
kafka_cluster_bootstrap_server_<cluster_type>
kafka_schema_registry_key_<cluster_type>
kafka_schema_registry_secret_<cluster_type>
kafka_schema_registry_url_<cluster_type>
spring_cloud_stream_kafka_binder_brokers_<cluster_type>
spring_kafka_properties_sasl_jaas_config_<cluster_type>
spring_kafka_properties_schema_registry_basic_auth_user_info_<cluster_type>

For example, a standard cluster creates kafka_cluster_api_key_standard, while an enterprise cluster creates kafka_cluster_api_key_enterprise.

Breaking Change: If upgrading from a version without cluster type suffixes, you'll need to either:

1. Migrate your applications to use the new secret names, or
2. Run terraform state rm for the old secrets and let Terraform create the new ones

By default, the module creates a new Confluent environment. You can also use an existing environment by setting create_environment = false and providing the environment_id.

module "confluent" {
  source = "github.com/extenda/tf-module-gcp-confluent"

  environment  = "my-new-environment"
  name         = "my-kafka-cluster"
  cluster_type = "enterprise"
  project_id   = "my-gcp-project"
}

module "confluent" {
  source = "github.com/extenda/tf-module-gcp-confluent"

  create_environment = false
  environment_id     = "env-abc123"
  environment        = "unused"  # Required but ignored when using existing environment
  name               = "my-kafka-cluster"
  cluster_type       = "enterprise"
  project_id         = "my-gcp-project"
}

Note: When using an existing environment, the module skips creating Schema Registry API keys and related secrets. The assumption is that Schema Registry credentials are already managed separately for the existing environment.

This module supports Private Service Connect for dedicated clusters. When enabled, traffic between your GCP applications and Confluent Cloud does not traverse the public internet.

module "confluent" {
  source = "github.com/extenda/tf-module-gcp-confluent"

  environment  = "production"
  name         = "my-kafka-cluster"
  cluster_type = "dedicated"
  project_id   = "my-gcp-project"

  private_service_connect = {
    enabled        = true
    gcp_project_id = "my-gcp-project"  # GCP project that will connect via PSC
  }
}

After applying the module, use the private_service_connect output to create GCP endpoints:

# The module outputs service attachments per zone
# Example: module.confluent.private_service_connect.service_attachments
# {
#   "europe-west1-a" = "projects/.../serviceAttachments/..."
#   "europe-west1-b" = "projects/.../serviceAttachments/..."
#   "europe-west1-c" = "projects/.../serviceAttachments/..."
# }

This module supports Private Link Attachment for Enterprise and serverless clusters. Unlike Private Service Connect (which requires dedicated clusters and manual GCP endpoint creation), PLATT automatically creates all necessary GCP-side infrastructure including the Private Service Connect endpoint and private DNS configuration.

Note: Private Service Connect and Private Link Attachment are mutually exclusive.

module "confluent" {
  source = "github.com/extenda/tf-module-gcp-confluent"

  environment  = "production"
  name         = "my-kafka-cluster"
  cluster_type = "enterprise"
  project_id   = "my-gcp-project"

  private_link_attachment = {
    enabled    = true
    network    = "projects/my-gcp-project/global/networks/my-vpc"
    subnetwork = "projects/my-gcp-project/regions/europe-west1/subnetworks/my-subnet"
  }
}

When enabled, the module creates:

1. Confluent Private Link Attachment - Reserves a PLATT endpoint in Confluent Cloud
2. GCP Static IP - Internal IP address for the PSC endpoint
3. GCP Forwarding Rule - The Private Service Connect endpoint
4. Private DNS Zone - Zone for <region>.gcp.private.confluent.cloud
5. DNS Wildcard Record - Routes all Confluent traffic to the PSC endpoint
6. PLATT Connection - Registers the GCP endpoint with Confluent

After applying, your applications in the specified VPC can connect to Kafka using the private endpoint automatically via DNS resolution.

This module supports creating a cluster link to replicate data from an existing source cluster. The created cluster acts as the destination, receiving data from the source cluster via a destination-initiated link.

This is useful for:

• Disaster recovery (DR) setups
• Data migration between clusters
• Multi-region replication

module "confluent" {
  source = "github.com/extenda/tf-module-gcp-confluent"

  environment  = "production"
  name         = "my-kafka-cluster"
  cluster_type = "enterprise"
  project_id   = "my-gcp-project"

  cluster_link = {
    enabled                   = true
    link_name                 = "source-to-dest-link"  # Optional
    source_cluster_id         = "lkc-abc123"
    source_bootstrap_endpoint = "pkc-xxxxx.us-central1.gcp.confluent.cloud:9092"
    source_api_key            = var.source_api_key
    source_api_secret         = var.source_api_secret
    mirror_topics             = ["orders", "events", "users"]  # Optional
  }
}

The module creates a destination-initiated cluster link, meaning the destination cluster initiates outbound connections to the source cluster. This works well when:

• The destination cluster uses private networking (PLATT) and the source cluster is public
• The destination can reach the source's bootstrap endpoint

When mirror_topics is specified, the module creates confluent_kafka_mirror_topic resources for each topic. These mirror topics automatically replicate data from the corresponding source topics.

[
  "tf-confluent-api-key",
  "tf-confluent-api-secret"
]