Publishing my Website using GitLab CI Pipelines

2022-02-08

1156 words, ~8min reading time

I wrote some posts recently, like Update on Publishing my Emacs Configuration, where I mention that my current workflow of deploying changes to my website can be improved. Well, I could always improve it, but this is one of the more urgent things.

The Status Quo

Currently after I writing some blog post or changing a page I export it by calling the relevant ox-hugo exporter using the Org export dispatcher. This places the exported files in the content directory. When I'm ready to publish I run my “trusty” script which removes the current public folder (the place where hugo dumps all its files), runs hugo to generate all files from scratch and uploads it with rsync.

There is just on problem with this approach. I'm often using a different environment than the last time to edit the site. Sometimes I use another laptop, sometimes another operating systems and sometimes even both. I don't want to switch them just for writing a blog post but I want to use what's currently running. For publishing the source code, working with multiple environments and not at last to have some version control keep my website in a Git repository. If you ever used Git with more than one machine you know that forgetting to pull before starting to work on something (or in even worse situations after making a commit) happens almost on a regular basis. While its no fun to deal with this, at least you realize it. Git will scream at you until you get it right.

But there's another thing that doesn't scream. That doesn't say one word: Blog posts and updated sites that are not exported don't scream. They are that quiet that I only notice it by chance if they are missing on the website after uploading my page. And belief me: this did not happen only once!

“But why don't you just include a script to export everything before publishing?”

Because it takes horribly long. I have over 100 blog posts and 366 posts from my Project 365 in 2015. So some other solution is obviously needed!

The new workflow

This “other solution” is called continuous deployment. Let me outline shortly what I want. While I host my Git repositories on my Gitea instance and only mirror to GitHub and GitLab I currently have no own continuous integration / pipeline runner (I tried Woodpecker but don't want to run it on my main server and I don't need it that much that it is worth renting another VPS). So I decided to use GitLab Pipelines for this. The pipeline will run on every push and thereby build and deploy the website.

The Export Script

For the build step I wrote a short Emacs Lisp script that I'll discuss in parts.

(package-initialize)
(add-to-list 'package-archives '("nongnu" . "https://elpa.nongnu.org/nongnu/") t)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)

(setq-default load-prefer-newer t)
(setq-default package-enable-at-startup nil)

(package-refresh-contents)
(package-install 'use-package)
(setq package-user-dir (expand-file-name "./.packages"))
(add-to-list 'load-path package-user-dir)
(require 'use-package)
(setq use-package-always-ensure t)

The first part (well, nearly half the script) installs and loads the necessary packages. I added the Non-GNU ELPA and MELPA as package archives since I most likely need packages from them in the future, although currently only need ox-hugo which is available on MELPA. I install and load the packages using use-package since in my opinion this provides a clean structure.

(use-package org
  :pin gnu
  :config
  (setq org-todo-keywords '((sequence
                             "TODO(t!)" "NEXT(n!)" "STARTED(a!)" "WAIT(w@/!)" "SOMEDAY(s)"
                             "|" "DONE(d!)" "CANCELLED(c@/!)"))))

Of course I load Org and also define my org-todo-keywords list. ox-hugo will respect this and only export posts that don't have a keyword or have a keyword from the done part (the entries after the | (pipe)). To be honest I'm currently not using this feature for published blog posts since posts with a to-do-state would be visible in the public repos anyway. But I wanted to write the script as general as possible.

(use-package ox-hugo
  :after org)

For using ox-hugo I'm using ox-hugo, duh...

(defun mmk2410/export (file)
  (save-excursion
    (find-file file)
    (org-hugo-export-wim-to-md t)))

Then I define a small function that opens a given file and calls the ox-hugo exporter which exports the complete content (all posts/pages) of the current file.

(mapcar (lambda (file) (mmk2410/export file))
        (directory-files (expand-file-name "./content-org/") t "\\.org$"))

And finally I run this function for every file in my content-org directory. Currently there are only three but who knows what will happen in the future.

The Pipeline Configuration

For the upload SSH configuration I followed the corresponding GitLab documentation.

I started by creating a new user on my server and—using that user—a new SSH ed25519 key pair. Then I added the public key to the ~.ssh/authorized_hosts file and granted the user rights to write to the root directory of my website. Afterwards I defined some necessary CI variables in GitLab for connecting with this user.

• $SSH_PRIVATE_KEY: The private key for uploading to the server.
• $SSH_KNOWN_HOSTS: The servers public keys for host authentication. These can be found by executing ssh-keyscan [-p $MY_PORT] $MY_DOMAIN (from a trusted environment, if possible from the server itself).
• $SSH_PORT: The port at which the SSH server on my server listens
• $SSH_USER: The user as which the GitLab CI runner should upload the files.

Using these variables I can now write my .gitlab-ci.yml pipeline configuration.

variables:
  GIT_SUBMODULE_STRATEGY: recursive

Since I keep my own hugo theme in an own repository and import it as a Git submodule I can ask GitLab to by nice and clone it for me.

before_script:
  - apk add --no-cache openssh
  - eval $(ssh-agent -s)
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
  - mkdir ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$SSH_KNOWN_HOSTS" | tr -d '\r' >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts

The script then continues with a lot of SSH voodoo. After installing OpenSSH and starting the ssh-agent I add the private key and the public server key as a known host.

build:
  image: silex/emacs:27.2-alpine-ci
  stage: build
  script:
    - emacs -Q --script .build/ox-hugo-build.el
    - apk add --no-cache hugo rsync
    - hugo
    - rsync --archive --verbose --chown=gitlab-ci:www-data --delete --progress -e"ssh -p "$SSH_PORT"" public/ "$SSH_USER"@mmk2410.org:/var/www/mmk2410.org/

Then it gets a little bit more obvious. Using the Emacs 27.2 Alpine Image by silex I already get the necessary Emacs installation and just need to run the Emacs Lisp script from above with it. Then I install the necessary dependencies for the next steps. First I build the page with hugo and finally upload the resulting public/ directory to my server using rsync. Thereby I define the ssh command with -e since there seems to be no other way to set a SSH port. Using the --delete option I also remove posts and files that I removed from the repo or that are no longer build.

artifacts:
  paths:
    - public

As a small gimmick I also publish the public directory of my website as a build artifact. There is no reason at all for this but I first started only building the blog a few days ago and didn't implement the deploy part until today. Maybe it will come in handy some day or I delete that part sooner or later.

You can find the complete files in my repository.

Next Steps

While Gitea currently has a mirror feature it runs on a timer and not after each push. This means that I would either wait quite some time for Gitea to push the changes to GitLab or trigger the sync manually using the web frontend. Currently I'm doing the second one but this is not a good solution. I currently think about going back to my own workflow by declaring a server-side Git post-receive hook for mirroring.

Another step is improving the gitlab-ci.yml file. Adding rules to only run the pipeline on pushes to the main branch and splitting the one step into a build and a deploy step are things that I want to do quite soon.

Finally I also need to decide whether to continue publishing my Emacs config using Org publish and the config.mmk2410.org subdomain or whether I want to use ox-hugo for exporting to the /config path. In the later case I would need to further adjust the pipeline configuration and otherwise I would need to write an own pipeline.

As always, I'll keep you posted!

Day 11 of the #100DaysToOffload challenge.