Skip to content
K8SaaS

Google Cloud Platform (GCP)

Deploying an K8SaaS cluster on the Google Cloud Platform will require to use the gcp provider. In order to do so, follow the next steps.

Requirements

Section titled “Requirements”

Before your first implementation, make sure to be able to setup all the required properties available in the configuration reference.

Domain name

Section titled “Domain name”

Make sure to properly prepare a domain name prior to the K8SaaS cluster becoming operational.

RBAC

Section titled “RBAC”

ArgoCD needs these service accounts and keys:

Terminal window
CLUSTER_NAME=$(yq .clusterName bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_ID=$(yq .gcp.project bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_NUMBER=$(gcloud projects describe "${GCP_PROJECT_ID}" --format="value(projectNumber)")


gcloud iam --project=${GCP_PROJECT_ID} service-accounts create ${CLUSTER_NAME}-argocd
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member="serviceAccount:${CLUSTER_NAME}-argocd@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --role=roles/secretmanager.secretAccessor --condition="title=k8saas-cluster,expression=resource.name.startsWith(\"projects/${GCP_PROJECT_NUMBER}/secrets/${CLUSTER_NAME}_\")"
gcloud iam --project=${GCP_PROJECT_ID} service-accounts keys create argocd-gcp-key-file.json --iam-account="${CLUSTER_NAME}-argocd@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --key-file-type=json
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_argocd-gcp-service-account >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_argocd-gcp-service-account --data-file=argocd-gcp-key-file.json || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_argocd-gcp-service-account --data-file=argocd-gcp-key-file.json

You also need to create these service accounts and keys:

  • external-dns
  • cert-manager

Using these commands, global variable can be extracted from your configuration file:

Terminal window
CLUSTER_NAME=$(yq .clusterName bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_ID=$(yq .gcp.project bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_NUMBER=$(gcloud projects describe "${GCP_PROJECT_ID}" --format="value(projectNumber)")
GCP_DNS_PROJECT_ID=$(yq .gcp.dnsProject bootstrap-config.yaml | tr -d '"')


gcloud iam --project=${GCP_PROJECT_ID} service-accounts create ${CLUSTER_NAME}-<service-name>
gcloud projects add-iam-policy-binding ${GCP_DNS_PROJECT_ID} --member="serviceAccount:${CLUSTER_NAME}-<service-name>@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --role=roles/dns.reader
gcloud dns --project=${GCP_DNS_PROJECT_ID} managed-zones set-iam-policy <your-dns-zone> \
--policy-file=<(gcloud dns --project=${GCP_DNS_PROJECT_ID}  managed-zones get-iam-policy <your-dns-zone> --format json | jq '.bindings += [{members:["serviceAccount:'${CLUSTER_NAME}'-external-dns@'${GCP_PROJECT_ID}'.iam.gserviceaccount.com"], role:"roles/dns.admin"}]')
gcloud iam --project=${GCP_PROJECT_ID} service-accounts keys create /dev/stdout --iam-account="${CLUSTER_NAME}-<service-name>@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --key-file-type=json | { \
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_<service-name>-gcp-service-account >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_<service-name>-gcp-service-account --data-file=- || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_<service-name>-gcp-service-account --data-file=-; }

You will also need to create apps oidc secrets and export to GCP Secret Manager for:

  • argocd
  • argo-workflows
  • grafana
  • oauth2-proxy
  • oauth2-proxy-cookie-key
  • s3gw-access-key
  • s3gw-secret-key
Terminal window
CLUSTER_NAME=$(yq .clusterName bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_ID=$(yq .gcp.project bootstrap-config.yaml | tr -d '"')


openssl rand -hex 16 | { \
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_<secret-key> >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_<secret-key> --data-file=- || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_<secret-key> --data-file=-; }

And finally, set your admin password:

Terminal window
CLUSTER_NAME=$(yq .clusterName bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_ID=$(yq .gcp.project bootstrap-config.yaml | tr -d '"')
admin_password=$(openssl rand -hex 16)


echo "$admin_password" | { \
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_admin-password >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_admin-password --data-file=- || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_admin-password --data-file=-; }


echo "$admin_password" | htpasswd -BinC 10 _ | head -n 1 | cut -d: -f2 | { \
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_admin-password-hash >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_admin-password-hash --data-file=- || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_admin-password-hash --data-file=-; }

OpenCost credentials

Section titled “OpenCost credentials”

OpenCost requires access to the Google Cloud Billing API to retrieve resource pricing. Without this API enabled, OpenCost will not function correctly with the default configuration.

Additionally, deploying on GCP does not guarantee that standard pricing applies. Internal company rates, custom discounts, or unaccounted pricing adjustments may impact cost calculations.

  1. Activate the Cloud Billing API

  2. Create and store the Cloud Billing API key

Terminal window
CLUSTER_NAME=$(yq .clusterName bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_ID=$(yq .gcp.project bootstrap-config.yaml | tr -d '"')


gcloud services api-keys get-key-string --format="value(keyString)" $(gcloud services api-keys list --filter="displayName=${CLUSTER_NAME}_opencost-access" --format="value(name)" --verbosity=error | grep . || gcloud services api-keys create --project=${GCP_PROJECT_ID} --api-target=service=cloudbilling.googleapis.com --display-name="${CLUSTER_NAME}_opencost-access" --format="value(response.name)" 2>/dev/null) | { \
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_opencost-api-key >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_opencost-api-key --data-file=- || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_opencost-api-key --data-file=-; }
  1. Create a service account, bind the needed policies and store the key
Terminal window
CLUSTER_NAME=$(yq .clusterName bootstrap-config.yaml | tr -d '"')
GCP_PROJECT_ID=$(yq .gcp.project bootstrap-config.yaml | tr -d '"')


gcloud iam --project=${GCP_PROJECT_ID} service-accounts create ${CLUSTER_NAME}-opencost
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member="serviceAccount:${CLUSTER_NAME}-opencost@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --role roles/compute.viewer --condition=None
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member="serviceAccount:${CLUSTER_NAME}-opencost@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --role roles/bigquery.user --condition=None
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member="serviceAccount:${CLUSTER_NAME}-opencost@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --role roles/bigquery.dataViewer --condition=None
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member="serviceAccount:${CLUSTER_NAME}-opencost@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --role roles/bigquery.jobUser --condition=None
gcloud iam --project=${GCP_PROJECT_ID} service-accounts keys create /dev/stdout --iam-account="${CLUSTER_NAME}-opencost@${GCP_PROJECT_ID}.iam.gserviceaccount.com" --key-file-type=json  | { \
gcloud secrets --project=${GCP_PROJECT_ID} describe ${CLUSTER_NAME}_opencost-gcp-service-account >/dev/null 2>&1 && \
gcloud secrets --project=${GCP_PROJECT_ID} versions add ${CLUSTER_NAME}_opencost-gcp-service-account --data-file=- || \
gcloud secrets --project=${GCP_PROJECT_ID} create ${CLUSTER_NAME}_opencost-gcp-service-account --data-file=-; }

GCP configuration example

Section titled “GCP configuration example”

Once ready, creating a config.yaml file with the next parameters will be required.

config.yaml
provider: gcp
region: <your-region>
overlays:
- base-gcp
- gcp
- gcp-dns
- gcp-tls


k8s:
  version: v1.29.0
  controlPlaneMachineCount: 1
  workerMachineCount: 1


image:
  name: ubuntu-2204-k8s-1-29-0 # qcow2 file


gcp:
  keyFile: gcp-key-file.json
  project: <project-name>
  controlPlaneMachineType: n1-standard-2
  nodeMachineType: n1-standard-4
  network: <network-name>


git:
  global:
    branch: main
    credentials:
      username: root
      password: Passw0rd
  bases:
    endpoint: http://gitea-http.k8saas-system.svc:3000/root/k8saas-bases
    repository: k8saas-bases
  config:
    endpoint: http://gitea-http.k8saas-system.svc:3000/root/k8saas-config
    repository: k8saas-config
  frozen:
    endpoint: http://gitea-http.k8saas-system.svc:3000/root/k8saas-frozen
    repository: k8saas-frozen
  custom:
    endpoint: https://github.com/ccl-consulting/k8saas-user-custom
    repository: k8saas-user-custom

QCOW 2 Image

Section titled “QCOW 2 Image”

To provide a QCOW2 image for your K8SaaS deployment on GCP, include the image name in your config.yaml file. This name should specify both the OS version and Kubernetes version, for example: ubuntu-2204-k8s-1-29-0. Ensure the image is correctly prepared and accessible for deployment.

SecretStore required values

Section titled “SecretStore required values”

Make sure to include your SecretStore configuration in your config.yaml file.

config.yaml
secretStore:
  provider: gcpsecrets
  gcpSecretsKeyFile: ./access-gcp-key-file.json
  valsPrefix: secretref+gcpsecrets://{projectName}/{clusterName}_
  valsSuffix: "?trim_nl=true"