RSS

Hugo with Docsy and AWS Amplify

How after 10 years I moved from Blogger to Hugo, with Docsy theme, and hosted it on AWS Amplify for the benefit of CI/CD.
Share this page:

Why move from Blogger

I’ve been blogging about Technology since 2010. Some you might even know the previous version of Mats Cloud. Blogger is all right, BUT - the speed of static site generators combined with the flexibility and universality of MarkDown are just too tempting, which is why I’ve been looking for a new platform.

There is another reason why I was eager to move to a new platform. For a while I’ve been using a standard blog approach, post by post, but this will change. I want to use Mats Cloud for two things:

  • Blogging, for which Blogger is still fine, I guess.
  • Help community learn about Cloud and DevOps in a non “let’s see these 200 hours of Videos and you’ll make 20 different Hello World apps”, and switch to a “come up with your idea, and let’s help you build Production Ready and learn everything step by step” kinda way. Basically, focus on simplicity and specific technology stack deep dive.

Having clear requirements tends to help choose the right technology.

Why I’ve chosen this technology stack

Why Hugo

I’m a huge fan of Go or Golang, and I’ve been using it for Web backend due to it’s speed, simplicity in writing and reading the code, and the way it handles “parallelism” which is perfect for micro services. Point - Hugo (over Jekyll and all others bloggers are lovin’). Also… some of my favorite bloggers, like Scott Lowe and Ivan Pepelnjak migrated their blogs to Hugo, and they’re obviously smarter then me, so if they say “jump”…

Some of the benefits of Hugo (google this… what impressed me most is how much Hugo bloggers LOVE Hugo. Also, there’s insanely many “create your blog in 3 minutes” blogs):

  • Hugo takes caching a step further and all HTML files are rendered on your computer. You can review the files locally before copying them to the computer hosting the HTTP server. Most notable benefit of this is - performance!
  • Hugo does not use PHP, it does not use a database, and it doesn’t handle dynamic content. Sure your content is dynamic during creation, but once you compile it, it is just a standard HTML website, meaning - incredibly fast.
  • Other features, like Menus, XML Sitemap Generation, RSS/Atom Feed Generation, Analytics (via Google Analytics), Comments (via Disqus), Multilingual/il8n, Custom Output Formats etc.

Why Docsy

Since I was looking for a platform to do both, Blog and Document my How to become a Cloud Ninja course, I needed a Hugo Theme that would support these both “out of the box”. I’m also a huge fan of tightly organized content (check out the left side of my “Become a Cloud Ninja” page)… and after reviewing so, so many themes (WOW there are a LOT of Hugo themes these days), I chose the top 3: Hugo Themes I’ve considered:

  1. Hugo ink theme
  2. Academic theme
  3. WINNER - Docsy by Google

Occasionally, I did find a bug or an improvement in the Docsy code, so now I’m officially a contributor of Docsy. Not too many commits at the moment, but don’t worry, one by one.

Why AWS Amplify

This was an easy choice, and had I known how much AWS Amplify ROCKS - it would have been even easier. With my final setup, my blog writing experience is:

  • My Mac is my Dev environment. I write my posts in VS Code and start the Hugo Server on my machine to preview real time in the browsed what’s happening.
  • My QA is a “dev” branch on GitHub, where I deploy “big changes” first, to make sure everything runs in order. When I push code into Dev, GitHub takes care of my CI, and AWS Amplify starts the pipeline, and deploys the new code to my QA Environment. You see why AWS Amplify is important?
  • Once everything is all right with the QA, I merge to Master, and AWS Amplify deploys the the code to “production”, which is… well, this site you’re reading right now.

So, after checking out the official Hugo hosting and deployment docs, my first idea was to:

I’ve tried to get some help on Twitter, and from what I saw everyone kinda did a similar thing. So while I was checking out how to do CI/CD, I saw a demo of AWS Amplify somewhere, and I just seemed so perfect for a static content with CDN and low effort CD that it blew my mind… so I first wanted to give it a test run, but once I’ve tested the sweet effortless CD, I guarantee it was the right choice.

Hugo, GitHub, Docsy and Amplify: simple setup and 5 “simple” issues

This was the moment where my theory got confirmed… you remember, the one where I claim that there’s too many blogs that teach you how to do “hello world”, but too few that help with the REAL deal.

Prerequisites

Just to make sure we’re on the same page, if you’re going to follow these instructions, you need:

  • To buy a MacBook. If you’re a Linux user - that’s fine. If you think Windows will do - get the hell out of my blog!
  • AWS Account. Don’t worry, in the Amplify Pricing guide you’ll see that you’re within the Free Tier as long as you don’t go over 1000 build minutes per month, host less then 5GB and serve less then 15GB. You’re probably good, trust me… Amplify does charge you for the underlying AWS Services, which are S3, CloudFront, optionally a few others, but you’re still most probably in the Free Tier.
  • GitHub Account. And basic Git skills… like how to clone, add, push, merge etc.

Deploy Hugo

Docsy says you need an extended version, check it out, so if you’re also doing a theme with a similar requirement, you need to execute the following:

Important: Before you proceed with the next step, make sure the Theme you’re using is compatible with the last Hugo version. At this moment, Docsy is NOT, so I had to downgrade later on both Amplify and my Mac, and… that is NOT what you want to be doing! Before you continue, make sure you’re good, Ok?

Issues 1 and 2 are both related to the non compatible hugo version in both, AWS Amplify and Docsy” and the details on how I detected and managed to solve them can be found in my previous blog post.

Issue 3: GitHub Sync: Since you need your repo to be sync-ed with your remote Git site, not Docsy-s example, if you try to do what Docsy recommends - Git will give you an error when you try to push the new changes. I’m not entirely sure why this is, but from the Logs, and from this blog post on linuxhints.com, it seems like that when we cloned the Docsy Example repo using the “-depth 1”, we escaped whats called the “shallow clone”, so to avoid this, my advice is to use this workaround (step by step instructions below):

  • Create a Hugo site.
  • Create your Git Repo.
  • Get out of that folder, and clone Docsy Example in the common “root” ($HOME/Documents in my case).
  • Copy the content of your Docsy Example to your folder.
  • Commit and push everything to “master”.

Install Hugo, and make sure your version is an “extended” one:

cd $HOME/Documents //or wherever you want to deploy your site...
brew install hugo

Omitted installation info…

hugo version
Hugo Static Site Generator v0.69.0/extended darwin/amd64 BuildDate: unknown

hugo new site matscloud
Congratulations! Your new Hugo site is created in /Users/matjovanovic/Documents/matscloud

Just a few more steps and you're ready to go:

1. Download a theme into the same-named folder.
   Choose a theme from https://themes.gohugo.io/ or
   create your own with the "hugo new theme <THEMENAME>" command.
2. Perhaps you want to add some content. You can add single files
   with "hugo new <SECTIONNAME>/<FILENAME>.<FORMAT>".
3. Start the built-in live server via "hugo server".

Visit https://gohugo.io/ for quickstart guide and full documentation.

Create a GitHub repository

Next go to your folder, and init the Git, but DON’T set the theme just yet (refer to the Issue 3, explained above). I’ll assume you’ve configured Git on your Mac. Go to your new hugo site folder, and create a Git repo that you will use for your Hugo web site:

cd matscloud
git init
git add README.md
git commit -m "first commit"
git remote add origin :YOUR_GIT_USER/REPO_NAME.git
git push -u origin master

Now, before you download Docsy stuff, make sure to follow THEIR instructions, and install PostCSS (commands below are for MacOS):

cd matscloud
sudo npm install -D --save autoprefixer
sudo npm install -D --save postcss-cli

Clone Docsy Example

Hugo Quickstart recommends you copy the Example web site, and I think you’ll make your life so much easier if you do the same, so be sure to clone the website, BUT in order to include all the folders and dependencies, include “recurse submodules”, as below:

git clone --recurse-submodules --depth 1 https://github.com/google/docsy-example.git

Ok, so now you’ve ended up with your ROOT hugo folder, and all your contents in the Docsy Example subfolder… let’s solve this (be sure you’re in the root folder of your website):

pwd
.../Documents/matscloud

ls
archetypes		content			docsy-example		node_modules		static
config.toml		data			layouts			package-lock.json	themes

Now is the time to remember all that stuff about the GitHub related Issue 3, which is why we now need to copy the content from the Docsy Example folder, into our website folder.

cp -a docsy-example/ .
git add .
git commit -m "docsy example added"
git push

Start the engines

All right, all ready to test your server:

hugo server

…and now bunch of stats about your web…

Ok, so this can work from the first attempt… or, if you’re a web freek like me, you’ll have a bunch of opened ports on your Mac, sand when you try to run Hugo, you’ll get the error, which brings us to the Issue 4:

ERROR 2020/04/21 16:02:17 Error: listen tcp 127.0.0.1:1313: socket: too many open files in system

Issue 4: too many open files in system: There are 2 ways to solve this, and whichever one you choose, you’re doing this on your own responsibility.

Solution to Issue 4, Option 1

You can check the values with:

sysctl kern.maxfiles
sysctl kern.maxfilesperproc

You can increase the limits (at your own risk) with:

sysctl -w kern.maxfiles=20480 (or whatever number you choose)
sysctl -w kern.maxfilesperproc=18000 (or whatever number you choose)

To make the change permanent, use sudo to put your settings in /etc/sysctl.conf (which you may have to create), like this:

kern.maxfiles=20480
kern.maxfilesperproc=18000

This should work, but before you decide, check out the Option 2, below.

Solution to Issue 4, Option 2

A elegant way to solve this was of course, found on StackOverflow. It was created for Sierra, but works on my Catalina.

What you need to do is edit the Max Files in the Launch Deamons config within the Library. Don’t forget to use the “sudo” or it won’t work. Also… this should emphasize that this is not a zero risk operation:

you need “sudo” to do “vi” in /Library if you’re on Catalina

cd /Library/LaunchDaemons
vi limit.maxfiles.plist

Add the following:

<?xml version="1.0" encoding="UTF-8"?>  
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"  
          "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">  
    <dict>
      <key>Label</key>
      <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
      <array>
        <string>launchctl</string>
        <string>limit</string>
        <string>maxfiles</string>
        <string>64000</string>
        <string>524288</string>
      </array>
      <key>RunAtLoad</key>
      <true/>
      <key>ServiceIPC</key>
      <false/>
    </dict>
  </plist>

Change the owner, load the new settings, and make sure everything is in order:

sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist
sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist
launchctl limit maxfiles

Ok, let’s try to run the hugo server again now:

hugo server

… various checks and stats…

Web Server is available at //localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Now go to your Web Browser, and type “localhost:1313”. Your first draft is ready!!!

Hosting and CI/CD with GitHub and AWS Amplify

As mentioned before, AWS Amplify was a very pleasant surprise for me. It’s perfectly safe to follow the official Hugo’s AWS Amplify Hosting instructions, but don’t forget to check out my post about Amplify and Hugo version issues before you start deploying.

At first, I only deployed the “production” version of the website, using the “master” branch. A few tips.

Tip 1: You need to authorize AWS to access GitHub

Be sure you’re doing this from the AWS Region you’ll be deploying your page. Once you’re done, you will be able to create other “environments”, such as QA or Test_Feature, this action is done only once.

Authorize GitHub

Tip 2: Learn to use QA and Prod

Once everything is up and running, Amplify will basically “listen” to every change on your Git repo master branch, and deploy a pipeline, including:

  • Provision content from your Markdown stored on GitHub.
  • Build the Hugo Server.
  • Deploy your web page using the randomly assigned Amplify DNS, or your own custom domain. Piece of advice: buy a domain, .com will cost you 12a month if you buy on AWS Route53, and that’s with SSL “as a card” by Amazon. You can do it all from AWS Console.
  • Verify that your content is displayed correctly on most common PC and Mobile screens (yeah… this is awesome!).

Full CI/CD is awesome, but it’s got certain risks. Have in mind that in this moment you’re operating in two environments:

  • Dev on your MacBook.
  • Prod, which is your master branch on GitHub.

You test your web on your PC using “localhost”. You run with your version of Hugo and other tools, and maybe it won’t run the same way in AWS, and if you’re committing to “master” directly, your actual web might be down or malfunctioning for hours till you find a way to manually do a rollback. This is why we use QA environments, to test everything before deploying to production. Just imagine how many changes you’ll be doing while configuring the theme, and migrating/adding the content… it’s just not worth the risk.

My suggestion is:

  • Use your Mac as Dev environment. Play around, test, and break stuff.
  • Add a “qa” or “dev” branch in GitHub as your QA environment, and add it to AWS Amplify.
  • Use Access List to protect your QA environment using user/password, so that only you can access, as shown below.

Access Control in Dev

Whenever you make a change follow the process:

  • Test on your Mac. always have “dev” or “qa” Git branch set on your folder.
  • If it works in dev - commit to “dev” on GitHub and push. AWS Amplify will do it’s magic in around 5 minutes, which is when you can check your QA URL.
  • If it works in QA - only merge to master.

If you set up everything correctly, this is how your “application” should look like on AWS Amplify:

Success in both environments

Migrating content from Blogger to Hugo

I’ve seen bunch of tools to migrate your content, but blog2md is the one I choose, basically because it also migrates comments and pictures, and - it’s awesome! It took it 3 minutes to migrate my blogger “backup” XML to a bunch of individual POST.MD files, and I did not have to do anything manually. All the instructions are pretty clear on the Git Repo, and it supports WordPress and Blogger.

You need to have NPM installed, as it’s a node project.

Integrate blog post comments section + Issue 5 - disqus integration

Comments section on your blog posts is something quite essential. Even so, I’ve decided NOT to migrate the comments from my Blogger blog. I know… sad. There’s another reason for you to comment on my posts :)

I’ve been using Disqus before, and it’s the one Hugo recommends, so - I just went for it. You need to create Disqus account, and it’s all fairly well documented on Disqus web.

This is where I’ve noticed that Docsy might be lacking users and a bit stronger community… and we’re talking about one of Googles opensource projects, and one of the most popular Hugo Themes. I followed the official instructions on Hugo website, and it all seemed pretty straight forward. I added the disqusShortname, Docsy Go templates should be made to include it automatically, but it simply didn’t work.

After infinite number of hours, I managed to find the issue, and it was due to Hugo change in parameter names. Basically, the parameter is now called differently, and all “content.html” files in the theme need to be updated. This is the change:

{{ if (.Site.DisqusShortname) }} - should be "Site.Params.DisqusShortname"

I did a pull request to an official Google Docsy repo, but until it’s merged, the workaround is to simply change the parameter names manually.