Dec 28, 2020

Playing with automatic deployment

Today everything tend to move to cloud; still, I wanted to make the experiment in hosting my own blog on my own hardware, in my home network, and then setting up a fully automated deployment flow – very much like GitHub pages.
I am not sure this will work out forever, as the hardware (and bandwidth) I have is not really able to sustain an heavy traffic; still it is for sure a nice learning experience. Besides, the ability to have a reverse proxy on my lan is a useful alternative to tools like ngrok.

Here is how I did it.

Self-hosting the website

There are few prerequisites here: a server that can host the blog, a router with possibility to configure the firewall, and a domain.

Generally speaking, this is the stack I have settled on:

GoDaddy for the domain. I am not really fond of GoDaddy. but this was the only option for using a personalized email address in outlook.com.
In my case, I had to add an A record for “rob” pointing to 77.237.25.148 in GoDaddy’s DNS management
A router with openwrt. I started with tomato, then migrated to openwrt. I like being in control of my network, and I like being forced to learn and know about networking.
Here, I added a firewall rule forwarding http and https traffic to the internal server. To be honest, I am not totally happy with this, but in the end I am exposing only static content, so it should be OK.
A home server, with ubuntu and nginx. Nothing really to say here, but this home server has already caused me a lot of unintended work. The site configuration was very simple, something like
```
server {
  server_name rob.liffredo.net;
  location / {
      root /data/www/rob.liffredo.net;
  }
}
```
A static site, made with Hugo (see later for details).
Letsencrypt for certificates. I am literally in awe for how they have simplified the management of an ssl certificate; I still remember how hard and expensive it was in the past. Now? Just literally five minutes.
```
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d rob.liffredo.net
```

At this point, I was able to see the default nginx message when going to https://rob.liffredo.net. Success!

Generate static content

The end goal is to have a git-managed flow: commit the change, push, and see the updates on the site.
However, this requires some tinkering on the CI side, so I want first to secure the manual process.
I am using hugo, but the workflow should be about the same with any other system.

Ensure to be in the correct timezone. Hugo will use it for displaying the time.
```
timedatectl
sudo timedatectl set-timezone Europe/Warsaw
```
Install hugo
```
sudo apt install hugo
```
Create access token from https://github.com/settings/tokens Everybody should use 2FA; but this will make things a bit less comfortable. In this case, I have to create a new token that will be used only here. I gave it repo permissions, but I suspect it might be enough to give something less.
[UPDATED - January 2024] Create a deployment key for github: Originally, I was recommending to clone the repo using https, because of issues using ssh; however, this approach is no longer working so I had to find a new approach – thanks to dylancastillo for the blueprint! Courtesy from []
```
sudo -u deployment
ssh-keygen -t ed25519 -C "roblog"
touch ~/.ssh/config
chmod 600 ~/.ssh/config
vim ~/.ssh/config
exit
```
and then add the content to the file:
```
Host github-roblog 
  HostName github.com 
  AddKeysToAgent yes 
  PreferredAuthentications publickey 
  IdentityFile ~/.ssh/id_ed25519
```
finally, add the key to github
Clone the repo, using the name set in the ssh config:
```
git clone git@github-roblog:rliffredo/roblog.git
```
The themes are added as submodules; as such, they need to be initialized.
```
git submodule init
git submodule update
```
Publish the site, and then copy on the destination (from directory with data)
```
hugo --cleanDestinationDir -d ../www/rob.liffredo.net
```
Check https://rob.liffredo.net

Automatic deploy

The general idea is quite simple:

Get notification from github on each push using a webhook
The webhook on our side will pull the repo and then build it again with hugo

I am going to use webhook to create an endpoint that will be sitting behind nginx, which will be in this case acting as a reverse proxy.
The endpoint will be working in a service and with its own user.
Note that the version available for ubuntu/debian is too old and has issues with secret in payload, so it’s better to download one from github.

The user will need to have its own home directory, because that is where we are going to store credentials for pulling the repo. The credentials are stored only once, and separated from the code that could (and should!) be committed in a repository.

Troubleshooting

If something goes wrong, these tools can be useful for troubleshooting the webhook:

netcat is a very nice tool to do anything on tcp/udp. In this case, it is possible to inspect the payload sent by github by using nc -l 9000
payload is hashed using sha. shasum is a utility to check that out, even if it might be a bit too much work.
curl is another swiss knife for everything related to http; in this case it is possible to test the basic of the connectivity using something like curl -X POST -k -i 'https://rob.liffredo.net/_hooks/deploy-roblog'
Permissions on all files for both directories (sources and target) must be granted to the deployment user, or there will be issues in update and deployment.

Step-by-step instructions

Here is a list of all the command actually used for the process described above; the actual content might need to be adjusted for a different setup or a different distro from what I used (Ubuntu).

# Create user
sudo adduser --system --disabled-login --group deployment
# Change ownership of the repo to the user, and ensure that password is stored
cd /data/roblog
sudo chown -R deployment:deployment .
sudo chown -R deployment:deployment /data/www/rob.liffredo.net
sudo -u deployment git config credential.helper store
sudo -u deployment git pull
# Setup webhook (see later for details)
sudo mkdir /data/webhook
sudo wget https://github.com/adnanh/webhook/releases/download/2.8.0/webhook-linux-amd64.tar.gz
sudo tar -zxvf webhook-linux-amd64.tar.gz webhook-linux-amd64/webhook
sudo mv webhook-linux-amd64/webhook /data/webhook
sudo rm -rd webhook-linux-amd64
sudo rm webhook-linux-amd64.tar.gz
sudo vim /data/webhook/deploy_roblog.sh
sudo chmod +x /data/webhook/deploy_roblog.sh
sudo vim /data/webhook/hooks.json
sudo vim /lib/systemd/system/webhooks.service
sudo systemctl enable webhooks
# Setup reverse proxy
sudo vim /etc/nginx/sites-available/rob.liffredo.net
sudo nginx -t
sudo systemctl restart nginx.service
# Test webhook - perform some deploy
sudo -u deployment /usr/bin/webhook -hooks /data/webhook/hooks.json --verbose
# If everything is successful, kill the testing process and start the service
sudo systemctl start webhooks

deploy_roblog.sh

#!/bin/sh

git pull -f -ff -q
hugo --cleanDestinationDir -d ../www/rob.liffredo.net

hooks.json:

[
  {
    "id": "deploy-roblog",
    "execute-command": "/data/webhook/deploy_roblog.sh",
    "command-working-directory": "/data/roblog",
    "pass-arguments-to-command":
    [
      {
        "source": "payload",
        "name": "head_commit.id"
      },
      {
        "source": "payload",
        "name": "pusher.name"
      },
      {
        "source": "payload",
        "name": "pusher.email"
      }
    ],
    "trigger-rule":
    {
      "and":
      [
        {
          "match":
          {
            "type": "payload-hmac-sha256",
            "secret": "some_secret",
            "parameter":
            {
              "source": "header",
              "name": "X-Hub-Signature-256"
            }
          }
        },
        {
          "match":
          {
            "type": "value",
            "value": "refs/heads/main",
            "parameter":
            {
              "source": "payload",
              "name": "ref"
            }
          }
        }
      ]
    }
  }
]

webhook.service

[Unit]
Description=Webhooks server
After=network.target

[Service]
Type=simple
User=deployment
Group=deployment
WorkingDirectory=/data/webhook
ExecStart=/usr/bin/webhook -hooks /data/webhook/hooks.json
Restart=on-abort

[Install]
WantedBy=multi-user.target