I’m looking to migrate our Jenkins pipelines over to GitLab CI/CD, specifically targeting a cloud-native setup. I'm trying to decide between using GitLab-hosted runners versus deploying our own self-managed runners on EC2 or GKE. What are the key steps to authenticate with cloud providers securely without hardcoding IAM keys in .gitlab-ci.yml? Also, how do you handle auto-scaling runners to keep costs down while ensuring pipelines don't sit in a queue for 10 minutes?
3 answers
Implementing GitLab CI/CD in the cloud starts with choosing your Runners. If you want zero maintenance, use GitLab-hosted runners. However, for cloud-native environments where you need access to private VPC resources, Self-managed runners on Kubernetes (GKE/EKS) are superior.
To authenticate securely, stop using static IAM Access Keys. Instead, use OpenID Connect (OIDC). You configure an OIDC trust between GitLab and your cloud provider (AWS/GCP), which allows your CI/CD jobs to fetch temporary, short-lived credentials using a JSON Web Token (JWT). This follows the principle of least privilege and eliminates "secret sprawl."
Great advice on OIDC, Jennifer. Regarding the cost and scaling issue Robert mentioned, have you looked into the Docker Machine executor or the Kubernetes executor
For a complete cloud implementation, don't forget to use GitLab's built-in Package and Container Registries. This keeps your build artifacts and Docker images within the same ecosystem, reducing the latency and data transfer costs of pulling images from external registries during the deploy stage.
I agree with Mark. Using the internal registry makes "Review Apps" much easier to set up. You can automatically spin up a dynamic environment in your cloud for every Merge Request, allowing stakeholders to see changes live before they hit production.
I’ve found that the Kubernetes executor is the gold standard for cloud-native GitLab CI. You can set up a "Runner Manager" in your cluster that spins up a new Pod for every single job and terminates it immediately after. To save money, run these on Spot Instances (AWS) or Preemptible VMs (GCP). Just make sure to use retry rules in your .gitlab-ci.yml in case the cloud provider reclaims the instance mid-job. This setup gives you nearly infinite scale with minimal idle cost.