How to deploy CKAN on GCP using Terraform

In the previous posts you have seen how to deploy CKAN on a Kubernetes cluster on platforms like AWS and Azure, as well as locally. Today you will see how to do that using GCP. When it comes to creating infrastructure on any cloud provider, first thing that everyone are thinking is Terraform, so you will need to have that also installed. While we are at that subject, let’s describe the commodities that a tool like Terraform provides.

Short intro to IaC

Infrastructure as Code is the process of more practical management and provisioning of cloud and IT resources via machine readable definition files. IaC enables creation, management, and destruction of compute resources by statically defining and declaring these in code.

Terraform

Terraform is an open source tool for building, changing, and versioning infrastructure safely and efficiently. It generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can then be applied to update infrastructure resources.

Prerequisites

Back to our theme for today. To successfully set things up you will need a few tools to be set up. Make sure that you have the following things set up and updated to suitable versions:

• Google Cloud Account

• terraform

• gcloud cli and make sure that you are logged in

• kubectl

Steps

Create a new project where you will deploy the instance(you can do it through GUI or terminal)

For example, if you do it through terminal, it would look something like this:

# The first command is for enabling autocompletion
gcloud alpha interactive

gcloud projects create 

Then set the project as a default one using:

gcloud config set project 

Create a new service account for Terraform to use when deploying

Create a new Service account to use for the deployment using:

gcloud iam service-accounts create terraform-deployment --display-name "Terraform test deployment" --project 

Find the e-mail for the Service account that you have created using:

gcloud iam service-accounts list

Get a credentials file for using the created Service account using:

gcloud iam service-accounts keys create sa-creds.json --iam-account 

Set the proper authorizations for the Service account using:

gcloud projects add-iam-policy-binding  --member=serviceAccount: --role=roles/editor

Create your directory structure for the Terraform files

mkdir -p environments//ckan_instance

For example my deployment will be in europe-west-1 so that is the value I have used in the place of DEPLOYMENT_LOCATION. Files can be structured multiple ways, but we will follow Terraform’s best practices for creating infrastructure files. Files that you will be needing inside the ckan_instance directory to have are:

• variables.tf

  variable "project" {
    # Here set the project id for the project that you have created
    default = ""
  }
  
  variable "region" {
    default = "europe-west1"
  }
  
  variable "zone" {
    default = "europe-west1-b"
  }
  
  variable "cluster" {
    default = "ckan-cluster"
  }
  
  variable "credentials" {
    default = "./sa-creds.json"
  }
  

  Most used variables that our project would be needing. If needed, you can add more here.

• main.tf

  resource "google_project_service" "container" {
    service = "container.googleapis.com"
  }
  
  resource "google_container_cluster" "primary" {
    name               = var.cluster
    location           = var.zone
    initial_node_count = 3
  
    master_auth {
      username = ""
      password = ""
  
      client_certificate_config {
        issue_client_certificate = false
      }
    }
  
    node_config {
      oauth_scopes = [
        "https://www.googleapis.com/auth/logging.write",
        "https://www.googleapis.com/auth/monitoring",
      ]
  
      metadata = {
        disable-legacy-endpoints = "true"
      }
    }
  
    timeouts {
      create = "30m"
      update = "40m"
    }
  }
  

  Purpose of the first resource is enabling the Kubernetes Engine API, while the purpose of the second one is creating the Kubernetes cluster.

• output.tf

  output "cluster" {
    value = google_container_cluster.primary.name
  }
  
  output "host" {
    value     = google_container_cluster.primary.endpoint
    sensitive = true
  }
  
  output "cluster_ca_certificate" {
    value     = base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)
    sensitive = true
  }
  
  output "username" {
    value     = google_container_cluster.primary.master_auth.0.username
    sensitive = true
  }
  
  output "password" {
    value     = google_container_cluster.primary.master_auth.0.password
    sensitive = true
  }
  

  Output are the variables from the Cluster resource creation that in a more complex environment would more often than not be needed.

• providers.tf

  provider "google" {
    credentials = file(var.credentials)
    project     = var.project
    region      = var.region
  }
  

  Basic definition of the Cloud provider for Terraform to know where and under which credentials it would be deploying.

Cluster creation

After the creation of the files the way that you did, cluster is ready to be deployed. You can do that using three commands from the directory where the .tf files are placed, in the example from the ckan_instance directory:

# Prepares the working directory for Terraform configuration
terraform init

# Dry runs the terraform apply command just in case for some existent errors
terraform plan

# Applies the configuration that is places in the current working directory
terraform apply

Cluster connection

After the successful creation of the cluster, you need to connect it to your kubectl configuration:

gcloud container clusters get-credentials ckan-cluster --zone europe-west1-b --project 

To verify that you are successfully connected to the cluster and it is the one that you are currently attached to, you can use:

kubectl config get-contexts

or:

kubectl config current-context

you should get something like this as an output for the second command:

gke__europe-west1-b_ckan-cluster

CKAN deployment

As a guide for deployment you can use a previous article on that subject which can be found here, precisely the Deployment section of it.

Note: Since the cluster is not local anymore, accessing the IP address from the outside(ex. your browser) will not work. For that to work also you will need to reserve a static address under the External IP addresses in the Platform console and then add it to the service.

Cleanup

Deleting everything that you did is as easy as it was creating the infrastructure. For that you can just use with ckan_instance as a current directory:

terraform destroy

Finishing note

We in Keitaro often use some additional configurations, like moving the PostgreSQL database out of the Kubernetes cluster to a Cloud SQL instance that the Google Cloud Platform provides for better and cleaner maintenance after the deployment, but I will leave that aside for now since it requires some more configurations.