GitLab CI — Improve your DevOps Experience

8 min readMay 3, 2021

Source: https://medium.com/dot-intern/integrasi-gitlab-ci-cd-dengan-heroku-pada-aplikasi-reactjs-188dfa4be35c

This article is written as a part of Individual Review of Fasilkom UI’s Software Engineering Project Course 2021

DevOps

Imagine you’ve built your codebase and ready to deploy it. In traditional software development model, who write code to be organizationally and functionally is apart from those who deploy and support that code. The two different teams: Development Team and IT/Ops Team have their own objectives. So initially, they’ll only consider their own things. What’ll happen? There may be some miscommunications between Dev Team and IT/Ops Team that can affect our business performance.

How about the different team started to merge or work together? Yes, you can. But, think how we can deliver the software faster with this software development model? In this era, we need to be more agile! Some of the testing and deployment process may often have the same procedure. How about we automate those process? That’s what you call “Improve your DevOps Experience with CI/CD”.

Deployment with CI/CD

A software that has been developed are going through some procedure — fast, automated, and reproducible manner —before it is delivered, which we call (CI/CD) Pipeline. The main concepts attributed to CI/CD are Continuous Integration, Continuous Delivery, and Continuous Deployment. The word “continuous” here represents a concept (practice), which we process our software’s frequent releases, automated processes, repeatable and fast processing, throughout the lifecycle of apps, from integration and testing phases to delivery and deployment. The word CI refers to Continuous Integration, while CD refers to Continuous Delivery and/or Continuous Deployment, which are related concepts that sometimes get used interchangeably.

**CI/CD Illustration.** Source: https://www.redhat.com/en/topics/devops/what-is-ci-cd

What is the difference among them?

Continuous Integration allows you to continuously integrate code into a single shared and easy to access repository. That means that every changes of new codes are regularly built, tested, and merged into shared repository. This is an answer of having too many branches that may conflict each other on repository. In simple way, this concept ensures that merged changes into shared branches doesn’t broke the whole app by validating it automatically with some testing: unit testing and integration testing.

Continuous Delivery means that changes by developer can be automatically tested and uploaded into repository, which can be deployed by the IT/Ops Team to the production. This concept makes sure that we can deploy the code with minimal effort, and improve visibility and communication between Dev Team and Business Team.

Continuous Deployment means that our changes in the repository can be automatically going through some process to be deployed in production, which can be used by customers. With this concept, it reduces the “slow” manual deployment process by the IT/Ops Team.

GitLab CI

GitLab CI is one of the many tools that can be used for CI/CD. If we use GitLab Repository, we can use its GitLab CI through some configurations to automate the whole CI/CD process, called Pipeline.

GitLab CI/CD is configured by a file called .gitlab-ci.yml on the root of our repository. When there are some push event that changes the code, this file will create a pipeline, with one or more stages, and run by Runner that can Integrate, Delivery, and Deploy our changes.

A GitLab CI consists of several stages and each stage contains several jobs. Jobs are the most fundamental element of a .gitlab-ci.yml file, which run specific task from a stage, with a Runner. We will see how to configure .gitlab-ci.yml and see how does it work in real implementation.

Implementation in Software Engineering Project Course 2021 Fasilkom UI

My team — Magic People — uses GitLab CI/CD to automate all of the continuous methodologies (CI & CD). Actually, there are two repositories for the frontend side and backend side. This example will use the backend side repository. First, we setup the .gitlab-ci.yml to specify three stages: test, report, and deploy.

The first stage, test, consists of running unit & integration tests and checking if there are some violation of the best practice with flake8 lint. This stage makes sure that our pushed or merged changes into specific branch doesn’t broke the whole app (Continuous Integration).

The Test Job and Lint Job both use python:3.7 image. Because the Test Job will produce a file called coverage.xml that will be used on the report stage, this artifact is kept for 30 minutes. The Test Job ensures that all of the requirements all already installed and the migration is works. After that, we will run the unit & integration tests and produce the report. The Lint Job does the same.

Test:
  image: python:3.7
  stage: test
  artifacts:
    expire_in: 30 mins
  paths:
    - coverage.xml
  before_script:
    - pip install -r requirements.txt
    - python manage.py makemigrations
    - python manage.py migrate
  when: on_success
  script:
    - coverage run manage.py test
    - coverage report -m
    - coverage xmlLint:
  image: python:3.7
  stage: test
  before_script:
    - pip install -r requirements.txt
  when: on_success
  script:
    - flake8 --exclude migrations,settings.py,enable_cors.py --count

The second stage is report stage. This stage checks if there are some bugs, vulnerability issues, code smells, duplicate codes, security issues, etc, with the help from SonarQube, by running sonar-scanner. SonarQube is an open-source platform for continuous inspection of code quality to perform automatic reviews. This stage runs on master branch and staging branch.

SonarScanner:
  image:
  name: sonarsource/sonar-scanner-cli:4.6
  entrypoint: [""]
  stage: report
  script:
    - sonar-scanner
      -Dsonar.projectKey=$SONARQUBE_PROJECT_KEY
      -Dsonar.host.url=$SONARQUBE_HOST
      -Dsonar.login=$SONARQUBE_TOKEN
      -Dsonar.branch.name=$CI_COMMIT_REF_NAME
      -Dsonar.scm.disabled=True
  only:
    - master
    - staging

The last stage is deploy stage. This stage will deploy our changes into production automatically (Continuous Delivery & Continuous Deployment). With this stage, our team can deploy our change easily into staging or production platform on different Heroku apps.

Deploy_Staging:
  image: ruby:2.4
  stage: deploy
  before_script:
    - gem install dpl
    - wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
  script:
    - dpl --provider=heroku --app=$HEROKU_APPNAME_STAGING --api-key=$HEROKU_API_KEY_STAGING
  only:
    - stagingDeploy_Master:
  image: ruby:2.4
  stage: deploy
  before_script:
    - gem install dpl
    - wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
  script:
    - dpl --provider=heroku --app=$HEROKU_APPNAME --api-key=$HEROKU_API_KEY
  only:
    - master

In real project, you can define your own stages and tasks as long as it is not a part of reserved keywords. You can build your own to suit your needs. However, stages with no jobs will be ignored.

If we put it all:

stages:
  - test
  - report
  - deployTest:
  image: python:3.7
  stage: test
  artifacts:
    expire_in: 30 mins
  paths:
    - coverage.xml
  before_script:
    - pip install -r requirements.txt
    - python manage.py makemigrations
    - python manage.py migrate
  when: on_success
  script:
    - coverage run manage.py test
    - coverage report -m
    - coverage xmlLint:
  image: python:3.7
  stage: test
  before_script:
    - pip install -r requirements.txt
  when: on_success
  script:
    - flake8 --exclude migrations,settings.py,enable_cors.py --countSonarScanner:
  image:
  name: sonarsource/sonar-scanner-cli:4.6
  entrypoint: [""]
  stage: report
  script:
    - sonar-scanner
      -Dsonar.projectKey=$SONARQUBE_PROJECT_KEY
      -Dsonar.host.url=$SONARQUBE_HOST
      -Dsonar.login=$SONARQUBE_TOKEN
      -Dsonar.branch.name=$CI_COMMIT_REF_NAME
      -Dsonar.scm.disabled=True
  only:
    - master
    - stagingDeploy_Staging:
  image: ruby:2.4
  stage: deploy
  before_script:
    - gem install dpl
    - wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
  script:
    - dpl --provider=heroku --app=$HEROKU_APPNAME_STAGING --api-key=$HEROKU_API_KEY_STAGING
  only:
    - stagingDeploy_Master:
  image: ruby:2.4
  stage: deploy
  before_script:
    - gem install dpl
    - wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
  script:
    - dpl --provider=heroku --app=$HEROKU_APPNAME --api-key=$HEROKU_API_KEY
  only:
    - master

Notice that there are some variables starting with ‘$’ sign. These are environment variables which can be configured from Settings ➔ CI/CD➔ Variables.

If all jobs are finished successfully, then the CI pipeline will show a green check.

Setup your own GitLab Runner

GitLab Runner is a build instance which is used to run the jobs over multiple machines and send the results to GitLab and which can be placed on separate users, servers, and local machine. There are two kind of runners: Shared Runners and Specific Runners.

Shared Runners are useful for jobs multiple projects which have similar requirements. By default, shared runners are provided by GitLab.

Specific Runners are useful if jobs have certain requirements or specific demand for the projects. This runners can be setup with our own machine and can be alternative runners if there are no shared runners available. In this example, I’ll demonstrate to setup a specific runner in our own machine with Docker.

First, create a gitlab-runner volume in your docker to save our runner’s configurations.

docker volume create gitlab-runner-config

Then, we will setup our runner with register command, and save it to our previously created volume (-v). We also tell the docker to run this gitlab-runner container interactively (-it).

docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register

Get the token from Settings ➔ CI/CD➔ Runners ➔ Specific Runners

Enter your configurations:

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.cs.ui.ac.id/Please enter the gitlab-ci token for this runner: 
(From Settings > CI/CD > Runners > Specific Runners)
xxxxxxxxxxxxxPlease enter the gitlab-ci description for this runner:
(Change with your own runner description name)
Runner MichaelPlease enter the gitlab-ci tags for this runner (comma separated):
<you can leave here empty>Please enter the executor: virtualbox, docker+machine, kubernetes, custom, docker-ssh, shell, docker-ssh+machine, docker, parallels, ssh:
dockerPlease enter the default Docker image (e.g. ruby:2.6):
alpine:latest

You can change the executor and default Docker image to suit your needs, in my backend project, I’ll use docker with alpine image to run GitLab CI tasks.

We will see that our Runner is already activated and will be used in our GitLab pipelines.

My Thoughts

CI/CD is an extremely useful practice because it can automate a repetitive and tedious task of code integration and software delivery/deployment. This concept solves integrity of merged branches (Continuous Integration), the miscommunication problem between Dev Team and IT/Ops Team (Continuous Delivery), and automate deployment to production (Continuous Deployment). This way can boost your DevOps experience!

I encourage you to use CI/CD in your project to be more agile. Thank you for reading and Happy Coding!