Introduction

I have been using AWS with Terraform to provision my development instance on AWS cloud. When I wanted to configure dotfiles, change SSH port and configure some cron jobs, I hit a bottleneck. I literally was not able to do those tasks with user scripts because they had no effect. That was the time when creating custom image of the distro came into my mind.

In this post I’ll go through the step which I took to create new image which has some pre-configured software and config which was really hard to configure with Terraform itself.

While you read this post, take a moment to connect with me on LinkedIn.

Even if you are not a developer, you might find this post useful if you want certain packages pre-installed and pre-configured on the EC2 operating system. So without further ado, let’s get started.

Pre-requisites

• AWS account. If you don’t have already, head over to https://aws.amazon.com/console/ and look for a button saying Create an AWS Account.
• You know how to create instances and connect to it via SSH.

I won’t be going through how to create an instance and connect to it in this post. But I have an advice. While creating instance keep in mind the size of the root block device. Keep it to minimum. Later on if you want to create an instance based on your AMI, the minimum storage you can allocate at that point of time should be more than the one used while creating the instance for AMI creation.

Instructions

I’ll use Amazon Linux 2 for the base image as I’m fan of Red Hat based distros. But if you know your way around your distro, there should be no problem.

Outline of packages/configs we are going to configure in this AMI.

• Install and configure basic packages
• Install dotfiles
• Change SSH port
• Create cron job for routine tasks

Install and configure basic packages

SSH into your instance and run the following command one by one. But before even you do that, I want you to run export HISTFILE=/dev/null; history -d $(history 1). I’ll tell you what this command does at the [end of the post].

sudo yum update -y -q

# Install initial tools
sudo yum group install 'Development Tools' -y -q
sudo amazon-linux-extras install epel -y
sudo yum install vim-X11 golang docker tmux tree python3 git-lfs htop -y

# Configure initial tools
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker ec2-user
sudo newgrp docker

# Set upper limit to journalctl
journalctl --vacuum-time=180d

Let’s go through above commands one by one.

sudo yum update -y -q

The yum update -y -q command updates the system, and dose not ask for confirmation (-y), and does it silently (-q).

sudo yum group install 'Development Tools' -y -q

This command install a group of packages tagged as Development Tools. Although I only use git from this group, having everything installed keeps things sorted. If you are tight on space, please consider only installing git with sudo yum install git.

sudo amazon-linux-extras install epel -y

Next we install epel which is a software repository which brings many other packages to yum’s doorstep.

sudo yum install vim-X11 golang docker tmux tree python3 git-lfs htop -y

These are the packages I use on regular basis.

• I use vim-X11 instead of vanilla vim for the mouse support. vim-X11 enables me to navigate windows with mouse.
• You probably should be aware of golang and docker if you read my blog on a regular basis.
• tmux enables me to use same ssh connection and multiplex multiple terminal window inside it.
• tree prints out directory tree in a hierarchical fashion. If you want to see it in action, go to your home folder and invoke it from there.

sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker ec2-user
sudo newgrp docker

With above commands, we start the docker service. We enable it on boot. Create a new group called docker. Add the ec2-user to the docker group. And then newgrp is used to change the current group ID during a login session.

journalctl --vacuum-time=180d

With this command, I intend to limit the log retention time to approx. 6 months.

Install dotfiles

I have a collection of dotfiles ranging in configuration of packages including

# Install dotfiles.
cd ~
git clone https://github.com/santosh/.dotfiles.git
cd .dotfiles
make install

The above commands will clone and install my personal dotfiles.

Change SSH port

I tend to change SSH port to something other than port 22.

sed -i.bak "s/#Port 22/Port 12121/g" /etc/ssh/sshd_config

Configure cron jobs

In this section, I’ll setup some tasks with crontab. Routine tasks include mostly the cleanup tasks to keep the EBS space free.

This is my entry in crontab:

#!bin/bash

# Monthly cleanup scheme for master_node of development environment.

# 1. Clean docker cache
docker system prune -f

# 2. Clean yum cache
sudo rm -rf /var/cache/yum

# 3. Clean /tmp (files older than 15 days)
find /tmp -ctime +15 -exec rm -rf {} +

I tent to run this monthly, so I have created an entry in crontab like this:

@monthly     /path/to/shell/script.sh

If you are not familiar with cron syntax, I find https://crontab.guru/ a great resource.

With this, my distro is ready to be transformed into an image or AMI which could later be used to boot up an cloud computer at AWS.

Here is the complete bootstrap file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

#!/bin/bash

# Note: This script is intended to run in an interactive shell. 
# Use this script to create fresh bootstrapped image.

{ set +x; } 2>/dev/null

sudo yum update -y -q
sudo yum upgrade -y -q

# Install initial tools
sudo yum group install 'Development Tools' -y -q
sudo amazon-linux-extras install epel -y -q
sudo yum install vim-X11 golang docker tmux tree python3 git-lfs htop -y -q
echo Done installing packages.

# Configure initial tools
sudo systemctl start docker
sudo systemctl enable docker

if [ $(getent group docker) ]; then
    echo "docker group already exists; skipping."
else
    sudo groupadd docker
fi

sudo usermod -aG docker ec2-user
sudo newgrp docker
echo Done installing docker

# Set upper limit to journalctl
journalctl --vacuum-time=180d

# Install dotfiles.
cd ~
git clone https://github.com/santosh/.dotfiles.git
cd .dotfiles
make install
echo Done configuring dotfiles.

# Change SSH Port

echo -n "ENTER a random port number: "
read SSH_PORT
if [[ ! $SSH_PORT =~ ^[0-9]+$ ]] ; then
    echo "SSH port number must be an positive integer."
    exit
fi
sudo sed -i.bak "s/#Port 22/Port $SSH_PORT/g" /etc/ssh/sshd_config

echo Done bootstrapping.

A more updated version can always be found at https://github.com/santosh/.dotfiles/blob/master/bootstrap/amazon-linux.sh. You can run this script in a single go with following command.

sh <(curl -s https://github.com/santosh/.dotfiles/raw/master/bootstrap/amazon-linux.sh)

In next section, we’ll see how

Create a boot image/AMI

0. Make sure you’re at EC2 home in AWS Console.

1. Select the instance in which you ran commands mentioned in previous sections.

2. Give a name for the AMI and hit Create Image.

3. This image can be used to boot up other instances.

Do you remember?

Do you remember I told you to run export HISTFILE=/dev/null; history -d $(history 1)? This command disables history preservation for the current shell session. In case you missed it, you might see the entered command if you press up arrow in bash. I wanted you to have fresh bash history when you start a new instance.

Conclusion

Do you find something missing? Is something broken? Please let me know by leaving a comment and we’ll fix it together.

If you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.