Terraforming Kubernetes

Terraform has a really nice Kubernetes interface. Gitlab has a docker registry and integrated CI. These features, combined, has enabled us to merge our kubernetes configs with my other terraform configuration that manages our infrastructure. This allows ops to manage applications and their deployment into my infrastructure in the same, single repository that describes ‘all the things’… load balancers, DNS, deployed applications, etc.

Here’s how I’m able to describe the connection to Kubernetes and the docker secrets in the Kubernetes cluster:

# initialize our provider
provider "kubernetes" {
  host                   = "api.aethereal.engineering"
  config_context_cluster = "aethereal.engineering"
}

# manage our docker credentials for gitlab in kubernetes
resource "kubernetes_secret" "docker-registry" {
  metadata {
    name = "docker-registry"
  }

  data {
    ".dockercfg" = <<EOF
{
  "registry.gitlab.com": {
    "username": "${var.docker_user}",
    "password": "${var.docker_pw}",
    "email": "${var.docker_email}",
    "auth": "${base64encode(format("%s:%s", var.docker_user, var.docker_pw))}"
  }
}
EOF
  }

  type = "kubernetes.io/dockercfg"
}

# variables
variable "docker_user" {
  default = ""
}

variable "docker_pw" {
  default     = ""
  description = "password or API token"
}

variable "docker_email" {
  default     = ""
  description = "password or API token"
}

These stanzas will initialize the kubernetes API connection and will create a docker registry secret which will allow kubernetes to connect to the docker registry where our site is hosted. In this case, we’re using the gitlab docker registry which is integrated with our source code management system (gitlab).

Next, I define my pod an service with references back to the kubernetes cluster provider above.

# manage my loadbalancer
resource "kubernetes_service" "www" {
  metadata {
    name = "www"
  }

  spec {
    selector {
      app = "${kubernetes_pod.www.metadata.0.labels.app}"
    }

    port {
      port        = 80
      target_port = 80
    }

    type = "LoadBalancer"
  }
}

# manage my pod
resource "kubernetes_pod" "www" {
  metadata {
    name = "www"

    labels {
      app = "www"
    }
  }

  spec {
    image_pull_secrets {
      name = "docker-registry"
    }

    container {
      image = "registry.gitlab.com/aethereal/www:1.0.5" # <-- change your app version here.
      name  = "www"
    }
  }
}

# add a CNAME for 'aethereal.io' pointing to the load balancer that kubernetes created.
resource "aws_route53_record" "www" {
  type    = "CNAME"
  zone_id = "${aws_route53_zone.io.zone_id}"
  name    = "www"
  ttl     = "3600"
  records = ["${kubernetes_service.www.load_balancer_ingress.0.hostname}"]
}

I can run this through terraform and it will connect to my kubernetes cluster and create a pod and a load balanced service for that pod which is listening on port 80. Kubernetes handles deploying the new container and creating and configuring load balancer for me. The last terraform stanza creates a ‘www’ CNAME so that my ‘www’ service is addressable via route 53 DNS.

That’s it. My whole workflow to push a new version of an app is:

  1. tag a new version of ‘www’ in the ‘www’ repository
    • gitlab CI creates a new docker image and stores it in the integrated registry
  2. increment the version of my application in my ‘infrastructure’ repo which hosts my terraform and push to master on ‘infrastructure’
    • gitlab CI runs terraform and updates (or creates) my kubernetes pods and service definitions for me

This workflow is great because:

  • my dev team can simply push and tag a new release of code.
  • my ops team can push the new version of code and manage the infrastructure/rollback/etc in the ‘infrastructure’ repo.
  • neither team requires actual access to AWS or Kubernetes to deploy code as the credentials are stored in secret variables in gitlab.

Hope this helps!

Phone

(612) 840-6253

Address

750 Margaret Street
Saint Paul, MN 55106
United States of America